Changes

[http://jenkins-ci.org/ Jenkins CI ] is used by Web QA to run automated tests. The web dashboard can be accessed [httphttps://selenium.qa.mtv2.mozilla.com:8080/ here] (VPN required). = Installation =Jenkins has been installed using the [http://mirrors.jenkinswebqa-ci.org/osx-stable/latest native Mac OS X installer]. It runs as a launch daemon, listening on port 8080. (If you want to know more about launchd and daemons, see [http://developer.apple.com/library/mac/#technotes/tn2083/_index.html here] and [http://developer.applemozilla.com/library/mac/#documentation/MacOSX/Conceptual/BPSystemStartup/Articles/LaunchOnDemandDaemons.html here].) = Configuration = == Boot configuration ==The following have been configured for when Jenkins boots: {| ! SETTING !! VALUE|- | heapSize || 2g|- | permGen || 512M|} * To view all settings in the file, run: <code>defaults read /Library/Preferences/org.jenkins-ci</code>* To get the value Details of a single setting, run: <code>defaults read /Library/Preferences/org.jenkins-ci SETTING</code>* To set the value of a setting, run: <code>defaults write /Library/Preferences/org.jenkins-ci SETTING VALUE</code> == LDAP authentication == We use LDAP Auth coupled with job-execution setup and job/system-config changes, so we know who did what, for easier tracking/troubleshooting To set it up:# Load http://selenium.qa.mtv2.mozilla.com:8080/configureSecurity/# Click the checkbox to "Enable Security"# Under "Access Control," select "LDAP" for the Security Realm# Enter the appropriate settings (see administration are available on our [https://bugzillamana.mozilla.org/show_bug.cgi?id=904287 bug 904287] for the values)# Now, choose "Logged-in users can do anything" from Authorization == Other configuration ==* Location of jenkins.war: /Applicationswiki/Jenkinsdisplay/jenkins.war* Location of Jenkins home directory: QA/Users/Shared/Jenkins/Home = Starting/restarting/stopping the service =* To restart via the dashboard visit [http://selenium.qa.mtv2webqa-ci.mozilla.com:8080/restart http://selenium.qa.mtv2.mozilla.com:8080/restartinternal Mana page]* To manually start the daemon: <code>sudo launchctl load /Library/LaunchDaemons/org.jenkins-ci.plist</code>* To manually stop the daemon: <code>sudo launchctl unload /Library/LaunchDaemons/org.jenkins-ci.plist</code>

 * [http://wiki.jenkins-ci.org/display/JENKINS/LDAPBuild-timeout+Plugin LDAPBuild Timeout] - User authenticationAbort builds that take too long* [httphttps://wiki.jenkins-ci.org/display/JENKINS/IRCConfig+File+Provider+Plugin IRCConfig File Provider] - IRC bot Centrally maintained configuration files for triggering builds and notificationsjobs

* [https://wiki.jenkins-ci.org/display/JENKINS/URLTrigger+Plugin URLTrigger] - Trigger builds based on the response of a URL* [http://wiki.jenkins-ci.org/display/JENKINS/Build-timeout+Plugin Build Timeout] - Abort builds that take too long* [http://wiki.jenkins-ci.org/display/JENKINS/Github+Plugin github] - Integrates githubGitHub

* [http://wiki.jenkins-ci.org/display/JENKINS/IRC+Plugin IRC] - IRC bot for triggering builds and notifications* [http://wiki.jenkins-ci.org/display/JENKINS/JobConfigHistory+Plugin Jenkins Job Configuration History] - Provides history of configuration changes* [httpshttp://wiki.jenkins-ci.org/display/JENKINS/thinBackup ThinBackupLDAP+Plugin LDAP] - Scheduled backups of system and job configurationsUser authentication* [httphttps://wiki.jenkins-ci.org/display/JENKINS/xUnitManaged+Script+Plugin xUnitManaged Script] - Generates reports based on test resultsCentrally maintained scripts for jobs* [https://wiki.jenkins-ci.org/display/JENKINS/SSHOwnership+Slaves+plugin SSH SlavesPlugin Ownership] - Allows you to manage slaves running on *nix machines over SSHShow job owners and co-owners* [https://wiki.jenkins-ci.org/display/JENKINS/MaskRole+PasswordsStrategy+Plugin Mask PasswordsRole Strategy] - Allows masking passwords that may appear in the consoleRole-based strategy for user management* [https://wiki.jenkins-ci.org/display/JENKINS/MultipleTest+SCMsstability+Plugin Multiple SCMsplugin Test Stability] - Allows a job to check out sources from multiple SCM providersHistorical information about the stability of tests

* [https://wiki.jenkins-ci.org/display/JENKINS/RebuildURLTrigger+Plugin RebuildURLTrigger] - Allows Trigger builds based on the user to rebuild response of a parametrized build without entering the parameters again* [https://wiki.jenkins-ci.org/display/JENKINS/Workspace+Cleanup+Plugin Workspace Cleanup] - Deletes the workspace before the build or when a build is finished and artifacts saved* [https://wiki.jenkins-ci.org/display/JENKINS/AnsiColor+Plugin AnsiColor] - Adds support for ANSI escape sequences in console output* [https://wiki.jenkins-ci.org/display/JENKINS/Priority+Sorter+Plugin Priority Sorter] - Add the ability to assign different priorities to jobs* [https://wiki.jenkins-ci.org/display/JENKINS/Email-ext+plugin Email-ext] - Extends Jenkins built in email notification functionality by giving you more control* [https://wiki.jenkins-ci.org/display/JENKINS/NodeJS+Plugin NodeJS] - Provides Jenkins integration for NodeJS & npm packagesURL

Several environment variables are configured in the [http://qa-selenium.mv.mozilla.com:8080/configure Configure System] page:

* '''PATH''' - The system PATH for binaries. The default is '''$PATH:/usr/local/bin'''.* '''PIP_DOWNLOAD_CACHE''' - Cache PIP downloads. The default is '''/Users/Shared/Jenkins${HOME}/.pip_download_cache'''.

* '''PYTEST_RERUNS''' - Number of times to rerun a failed test (used by [https://pypi.python.org/pypi/pytest-rerunfailures pytest-rerunfailures]). The default is '''25'''.* '''PYTEST_TIMEOUT''' - Timeout (in seconds) for each test (used by [https://pypi.python.org/pypi/pytest-timeout pytest-timeout]). The default is '''300''' (five 5 minutes). = B2G Nodes =* See'''SAUCE_CAPABILITIES''' - Single capability to set for all desktop [http: [[QA/Execution/Web_Testing/Automationsaucelabs.com/B2G_Nodes|B2G Nodes]Sauce Labs]jobs. = Upgrading =When upgrades are available they can be automatically installed from the * '''SAUCE_CAPABILITIES_MOBILE''' - Single capability to set for all mobile [http://selenium.qa.mtv2.mozillasaucelabs.com:8080/manage web dashboardSauce Labs]jobs. = Disaster recovery = == Backing up ==Jenkins is entirely file based so a true full backup would be simply storing a copy of the entire home directory in another location. That said, the [https://wiki.jenkins* '''SAUCE_FIREFOX_VERSION_MAC''' -ci.org/display/JENKINS/thinBackup ThinBackup plugin] is used to perform scheduled (daily) backups of the Jenkins system and job configurations The default Firefox version to use for OS X [http://fs2saucelabs.office.mozilla.org/public/QA/WebQA/Jenkins/ http://fs2.office.mozilla.org/public/QA/WebQA/Jenkinscom/Sauce Labs]jobs. Note that this backup does * '''notSAUCE_FIREFOX_VERSION_WINDOWS''' include plugins. == Restoring from backup ==If you are simply restoring system or job configuration then you can - The default Firefox version to use the backups created by the for Windows [httpshttp://wikisaucelabs.jenkins-ci.org/display/JENKINScom/thinBackup ThinBackup pluginSauce Labs]jobs. == Migrating to a new server ==If you are recovering from a hardware failure or are migrating to a new server then the backups created by the * '''SAUCE_WINDOWS_VERSION''' - The default Windows version for [httpshttp://wikisaucelabs.jenkins-ci.org/display/JENKINScom/thinBackup ThinBackup pluginSauce Labs] will not be adequatejobs. If you have access to * '''SELENIUM_HOST''' - The host running the original server then the best way to restore is to simply copy the entire Jenkins home directory to the new remote Selenium serveror hub. Make sure that the instance on the original server The default is stopped before performing the copy. If for some reason you do not have access to the original server then a partial backup will have to do. Install the [https://wiki.jenkins'''selenium-cihub1.org/display/JENKINS/thinBackup ThinBackup plugin] to a clean Jenkins instance and restore from the latest backupqa. Any plugins that are needed will have to be installed manuallyscl3. = Console =Log messages can be seen by opening the Console applicationmozilla. A log query named Jenkins CI has been created, which filters by the sender: <code>org.jenkins-ci</code>com'''.

== Upgrading == === Automatic upgrade is failing or not available ===This may be because the location of the jenkins.war file has the wrong permissions. These can be reset using:  sudo chown -R daemon /Applications/Jenkins/ sudo chmod a=rw /Applications/Jenkins/jenkins.* == Backups == === Cannot perform a backup ===If you see the messages in the Jenkins log then it's likely the that remote volume for the backup destination is not mounted.* Cannot perform a backup. Please be sure jenkins/hudson has write privileges in the configured backup path {0}.* Could not create backup directory. Backup cannot be performed. To fix this, connect to qa-selenium.mv.mozilla.com, and execute the following from a terminal (you can use SSH or VNC):  mkdir /Volumes/public sudo mount -t smbfs //guest@fs02/public /Volumes/public == Jobs == === General issues ===

* Enable verbose PIP output in the [http://selenium.qa.mtv2.mozilla.com:8080/configure Configure System] page to determine if installation of Python packages is causing issues. === PIP index is unavailable ===We use the main PyPI index for installing Python packages. If this experiences issues then we can enable use of PIP mirrors in the [http://selenium.qa.mtv2.mozilla.com:8080/configure Configure System] page. Alternatively, we can temporarily change the PIP index URL to our internal mirror: '''<nowiki>http://pypi.build.mtv1.svc.mozilla.com/simple</nowiki>'''. === PIP installation is slow ===If Python packages are slow to install you should check that PIP mirrors are not enabled in the [http://selenium.qa.mtv2.mozilla.com:8080/configure Configure System] page. If mirrors are not enabled, it may be that the main PyPI mirror is slow to respond. In this case, try temporarily changing the PIP index URL to the internal mirror: '''<nowiki>http://pypi.build.mtv1.svc.mozilla.com/simple</nowiki>'''. === Stalled node jobs ===If jobs running on nodes appear to have stalled in post-build actions it may be that the node momentarily lost connection with the master. We have seen this with our Linux nodes running Open JDK. The problem was resolved by switching to Oracle's JDK. To do this, take the node temporarily offline and SSH into the box. Then run the following commands to remove Open JDK and install Oracle's JDK:  sudo apt-get purge openjdk* sudo add-apt-repository ppa:webupd8team/java sudo apt-get update sudo apt-get install oracle-java7-installer You can confirm the version of Java using the command:  java -version

Once == PIP index is unavailable ==We use the main PyPI index for installing Python packages. If this is done, you experiences issues then we can launch the node again from Jenkins web consoleenable use of PIP mirrors.

= References = PIP installation is slow ==Some content has been taken from [https://wiki.jenkins-ciIf Python packages are slow to install you should check that PIP mirrors are not enabled.org/display/JENKINS/Thanks+for+using+OSX+Installer here]If not, it may be that the main PyPI mirror is slow to respond.