Penelope Developer Page

Building Penelope

Start to become familiar with the process by building Thunderbird. Go fetch and build Thunderbird. Start learning how it works.

Linux

These were the steps followed to build Thunderbird and Penelope from CVS on linux. These instructions assume an up to date linux machine. The machine I used is running Gentoo Linux 2006.1. More Thunderbird linux build details can be found at developer.mozilla.org

• Create a new directory and cd into it

• cvs -d :pserver:anonymous:anonymous@cvs-mirror.mozilla.org:/cvsroot co -r THUNDERBIRD_2_0_0_6_RELEASE mozilla/client.mk

• cvs -d :pserver:anonymous:anonymous@cvs-mirror.mozilla.org:/cvsroot co -r THUNDERBIRD_2_0_0_6_RELEASE mozilla/mail/config/mozconfig

• cvs -d :pserver:anonymous:anonymous@cvs-mirror.mozilla.org:/cvsroot co mozilla/penelope mozilla/extensions/penelope

• cvs -d :pserver:anonymous:anonymous@cvs-mirror.mozilla.org:/cvsroot up

• cd mozilla

• make -f client.mk checkout MOZ_CO_PROJECT=mail

• Create ~/.mozconfig with:

   . $topsrcdir/mail/config/mozconfig
   ac_add_options --enable-default-toolkit=gtk2
   ac_add_options --enable-xft
   ac_add_options --disable-tests
   ac_add_options --enable-extensions=default,penelope

Note that it is this last line that includes the building of the Penelope extension in the main build. The source code for Penelope can be found in mozilla/penelope, but the extension option looks inside mozilla/extensions/penelope

• make -f client.mk build

• Go get breakfast/lunch/dinner because it takes a while to do a full compile.

OS X

The steps to build Penelope from CVS on OS X are almost identical to the instructions for Linux. There are may caveats depending on whether you want a universal binary, which version of OS X you have, which version you want to target, etc. More Thunderbird OS X build details can be found at developer.mozilla.org.

The only deviation from the Linux directions is the installation of LibIDL and GLib, and contents of the ~/.mozconfig file.

• Rememer to do the following:

   chmod +x ./configure
   chmod +x ./build/autoconf/mozconfig*

• Install LibIDL (via orbit) and GLib using fink:

   $ sudo apt-get update
   $ sudo apt-get install orbit orbit-dev glib

• Create ~/.mozconfig with:

   export CC=gcc-3.3
   export CXX=g++-3.3
   . $topsrcdir/mail/config/mozconfig
   ac_add_options --enable-prebinding
   ac_add_options --disable-tests
   ac_add_options --enable-extensions=default,penelope

Making universal binary builds on OS X

Universal binaries are executables for Mac OSX that contain both PowerPC and Intel processor binaries. That way you can distribute just one application and installer that will work on both types of processors.

Be sure that you can successfully build for just one processor using the steps above. That way you won't be confused if a problem arises and not know whether the issue is with building in general, or with universal binaries in particular.

The Mozilla Developer center has some documentation that you will need to follow. In particular there are a couple of things worth pointing out as they can cause problems if not set up correctly:

• Make sure you have at least version 2.4.1 of Xcode due to a bug in Xcode 2.4. You have to be a Apple Developer Connection member, but that's free and only requires a valid email address.
• Install the 10.2.8 SDK from the Xcode disk image. If you've already installed XCode, then you can easily add the 10.2.8 SDK by running the CrossDevelopment.mpkg package from the Packages folder of the Xcode disk image. The 10.2.8 SDK doesn't get installed by default, so you'll have to do a "Customize" install and select the 10.2.8 SDK.
• You must use the MOZ_OBJDIR option in your .mozconfig file in order to place the generated files outside of your mozilla source tree. This is because both PPC and Intel binaries need to be created and must be kept separate. The build process will create two directories (ppc and i386) in your MOZ_OBJDIR directory, and the universal build will be placed in ppc/dist/universal with a symbolic link at i386/dist/universal.

Windows

See the steps above under Linux for how to get the source code via CVS.

Penelope is back to using the trunk version of the Mozilla platform, which uses the MozillaBuild environment. Make sure you go through in detail the setup documentation. It's pretty simple to set up with MozillaBuild now. You just download the package and the installer gives you all the right versions of the tools you need to build. You have to use their MSys shell (which is basically bash) in order for it to compile correctly. It's easiest just to use their batch files for starting up the shell and getting the environment set up right. I've only tried it with Visual Studio 2005 (the official version of VS for the trunk), which has a corresponding batch file of start-msvc8.bat.

Here's the options for your ~/.mozconfig file in order to make a debug build:

   . $topsrcdir/mail/config/mozconfig
   ac_add_options --disable-optimize 
   ac_add_options --enable-debug 
   ac_add_options --disable-static --enable-shared
   ac_add_options --disable-tests
   ac_add_options --enable-extensions=default,penelope

Finally, go to the root of the source tree and do a:

• make -f client.mk build

Windows Vista

Windows Vista provides some roadblocks in compiling Penelope. The first is the new security model. By default, administrator accounts under Vista run applications at a reduced privilege level called Standard User. Microsoft found that the most common legitimate reason to need administrator rights was when installing an application, and in order for all the existing installation programs to work under Vista they had to come up with a heuristic workaround. What Vista does is automatically attempt to elevate the privilege of any application that has the words "install" or "setup" in the name of the executable filename. I say "attempt" because there appear to be some situations where the privilege elevation fails, and one of those happens to be when called inside of scripts. I think it has to do with the way the process is created (privilege elevation happens when you call ShellExecute(), but not when CreateProcess() is used), but I'm not sure of that entirely because I don't know how MSys shells start up new processes.

And wouldn't you know it, one of the utilities used to compile Penelope falls in to this heuristic: nsinstall.exe. All that command-line tool really does is copy some files, which doesn't require Administrator privileges as long as you have have write access to the destination directory, but Vista's automatic privilege elevation thinks it might due to the name. The privilege elevation fails and so the build fails. Happy, happy, joy, joy.

There's two current workarounds. One is that the Command Prompt window that you use to compile Penelope can be run as Administrator. You can create a shortcut setting that up (its in the Properties, Compatibility tab), or you can right-click on a link to a Command Prompt (e.g. one that shows up on the Start menu) and select "Run as Administrator". This is an unsatisfying solution because it raises the privilege level unnecessarily, but it does work. It does happen to solve other permission-related issues with files in general, so it is an easy way to fix the problem. It's also no different than the security model of Windows before Vista (well, at least the way that most people set it up, which is to run as Administrator all of the time).

The other way is to tell Vista that the app doesn't really need Administrator privileges. You can do this via a manifest file, which is an XML file which has properties about the application. It has the same name as the executable (including the ".exe"), but with an extra extension of ".manifest" to it. The manifest file contents should look like this:

   <?xml version="1.0" encoding="UTF-8" standalone="yes"?>
   <assembly xmlns="urn:schemas-microsoft-com:asm.v1" manifestVersion="1.0">
     <assemblyIdentity version="1.0.0.0"
        processorArchitecture="X86"
        name="nsinstall"
        type="win32"/>
   
   
     <description>Description of your application</description>
     <trustInfo xmlns="urn:schemas-microsoft-com:asm.v3">
       <security>
         <requestedPrivileges>
           <requestedExecutionLevel
             level="asInvoker"
             uiAccess="false"/>
           </requestedPrivileges>
          </security>
     </trustInfo>
   </assembly>

The pertinent part that makes it work is the level="asInvoker" attribute, which tells Vista that it doesn't need any extra privileges to run. There's one snag to this, which is that if you've already run nsinstall.exe without the .manifest file, then Vista has already put the perceived needed privilege elevation in the fusion cache. In order to clear the cache you need to reboot Vista.

There is a third way to workaround the problem, but I don't like the long-term implications of it. You could rename the nsinstall.exe utility to something that doesn't have "install" or "setup" in the name and then change the build scripts to that effect. However, that means keeping a branch of the build scripts forever and that would probably cause more headaches down the road.

Reducing Build Time

The Thunderbird folks advise building in a subdirectory or subdirectories when making a change to the code to reduce the amount of time needed to build. By a narrowing down the build to at least the component you need to rebuild you can cut the compile time dramatically. You can cut down the build time even further by specifically providing the change logic yourself and rebuilding only those portions of the code that you know you changed.

If you use a separate object directory (as the default config suggests), then you'll need to look for the individual makefiles in the object directory hierarchy (for example in my setup I found them under "thunderbird-static" since that's my object directory name). Otherwise the makefiles will be alongside the source.
Scott MacGregor notes:

I don't use the objdir technique, but some developers really like it.
Most of those folks like to use it because they are building multiple
products like firefox and thunderbird and they only want to have one
source tree directory. They can just point their thunderbird build at
one objdir directory and the fx build at another, while working out of
just one instance of the source tree. But I mostly just build thunderbird,
so moving the objdir out of my source tree doesn't help me much.

Specific Example

A lot of the core functionality is provided in mail.dll, which is built from the mailnews directory.

For example, say I wanted to make a change to a portion of the IMAP code (e.g. I want to modify "nsIMAPNamespace.cpp").

Slow Method

The really slow method to recompile would be to invoke:

make -f client.mk build

in the root level of the mozilla source code. On my laptop, even with no changes to the source code, this takes over 12 minutes to complete.

Faster Method

A faster way to recompile would be to invoke "make" inside the mailnews directory (no arguments are needed for "make" since the mailnews directory contains "Makefile"). Without an object directory specified that would be mozilla/mailnews; with an object directory specified that would be something like mozilla/objectdir/mailnews.
On my laptop, this took less than 2 minutes to recompile.

Fastest Method

The fastest way to recompile would be to first invoke "make" inside the mailnews/imap directory.
Then invoke "make" inside the mailnews/build directory (which just re-links mail.dll).
On my laptop, this took less than 10 seconds to recompile (ignoring time for me to type, change directories, invoke the 2nd command etc.)

Troubleshooting

Here are some problems that have cropped up during development:

• The prefs.js file in the profile directory may have stale settings in it which override the behavior you are trying to test. You must remove them manually.
  • On Linux the profile directory is in ~/.thunderbird/
  • On Mac OSX the profile directory is in ~/Library/Thunderbird/Profiles/
  • On windows the profile directory is C:\Documents and Settings\<Windows login/user name>\Application Data\Thunderbird\Profiles\
• The extensions subdirectory may contain the penelope extension if it was installed for Thunderbird. This will conflict with an install of Eudora because the penelope extension resides in the Eudora install directory. Both extensions will be applied at the same time which will cause problems. If you install Eudora, you must remove any previous install from the profile directory
• Beware cross-volume symlinks (see Discussion page)
• Very bad things happen when you have an unresolved entity (missing a DTD file or entry in a DTD file). This can manifest as the whole application is a window with zero size (which is very hard to find!).
• Multiple key mappings occur when they are scoped in different keysets. When redefining a key mapping, make sure it is done in the correct keyset ID.
• Some key mappings canot be set via the prefs file. They seem to be dynamically updated via a xul overlay at run time. The only way I have found to redefine these keys is to create an overlay that remaps the keys.
• If the resulting build in dist/bin will not run try adding execute permission to dist/bin/run-mozilla.sh

• To re-create the Makefiles you have to remove the generated configure files then re-run make:

   rm config.log config.cache config.status mozilla-config.h
   make -f client.mk build

Debugging the Build Process

Trying to figure out what went wrong with a build can be tricky. Here's a tip for helping troubleshoot builds.

The -w flag will output a diagnostic line every time you enter and leave a directory. So it gives you a context of where the following commands are coming from. It also gives a nesting level. This flag is on by default with the Cygwin version of make on Windows, but needs to be added for Mac and Linux. To add it, either add this line to your .mozconfig:

   mk_add_options MOZ_MAKE_FLAGS=-w

or set the MAKEFLAGS environment variable to be -w.

Building Installers

Once you're done building the application, you also have the option of building the installer. Instructions are here, but they basically just say to cd in to MOZ_OBJDIR/mail/installer, and do a make from there (or make installer for Windows). Also for Windows, the instructions note there are some extra tools needed to build the self-extracting installer. For Mac universal binary builds, do the make from MOZ_OBJDIR/ppc/mail/installer. After building it, the Windows installer can be found in MOZ_OBJDIR/dist/install/sea/. The Mac disk image will be in MOZ_OBJDIR/ppc/dist/.

Customizing Mac Disk Images

The Mac installer is a disk image. When you open it, it mounts a volume that holds the application. It first shows a license agreement (EULA) and requires the user to agree to the license before proceeding. It then shows a folder containing the application, and the user needs to drag the application to the Applications folder in order to install it. The appearance of that folder is controlled by the .DS_Store file in the folder, just like all folders in the Finder. .DS_Store holds the window position and size, the size of icons, the arrangement of the files in the folder, the background image, and other properties. To modify these properties you just have to modify the .DS_Store file.

There are some web sites out there that mention you can create a dummy disk image, muck with the folder window in it, and the save the resulting .DS_Store file to reuse in another disk image. Turns out that for compatibility on other machines this doesn't always work. Evidently the Finder can create .DS_Store files that will not work when moved to a different machine.

What the Mozilla platform uses is a .DS_Store file that was edited by hand and known to be compatible. This file is named dsstore and is kept along with the branding. The build process for the disk image copies $(DIST)/branding/dsstore in to the disk image as .DS_Store.

Mark Mentovai is the developer who came up with the Mozilla disk image process, and he reverse-engineered the .DS_Store file format to a fairly complete degree. His notes for that can be found here. It's very helpful for figuring out where to make manual changes in the dsstore file.

Note that if you want to change the background image all you have to do is check in a new background.png file, and no changes to the dsstore file will be necessary. The most likely changes someone will need to make to the dsstore file in the future will be to window size and/or location.