ReleaseEngineering/PuppetAgain/HowTo/Build DEBs: Difference between revisions

← Older edit

ReleaseEngineering/PuppetAgain/HowTo/Build DEBs (view source)

Revision as of 02:08, 17 November 2015

5,058 bytes added , 17 November 2015

→‎Create pbuilder images

Rail

Confirmed users

3,104

edits

@@ Line 1: / Line 1: @@
+Within PuppetAgain, Debian packages are built from an expanded <tt>debian/</tt> directory kept in version control and an <tt>.orig.tar.bz2</tt> kept in the data directories.  This surfaces changes to the debian build scripts as diffs in the puppetagain repo, although it also requires some custom build scripts.
+Those build scripts, however, are implemented on hosts with the <tt>toplevel::server::pkgbuilder</tt> class.  This class helpfully creates a cowbuilder image and provides the <tt>pupppetagain-build-deb</tt> script to build a package from a debian directory.
+= Upstream Upgrades =
+If the new package requires lots of other packages that aren't in our repo, then that would be a problem and we should probably find a way to recompile instead.  But if just the desired package plus one or two others needs to be upgraded, then it should be OK.
 = Building =
-At the moment we don't have any shared resources to build Debian packages. You need to use Ubuntu 12.04 based VM on your laptop.
+Log in to a package builder host (a node that includes toplevel::server::pkgbuilder; see your org's nodes.pp to find the hostname) as yourself (not root), and get a checkout of the puppetagain repo. See [[ReleaseEngineering/How_To/Start_RPM_Ubuntu_Packager_Instances|moco instructions]].
+For the package you want, find or create the appropriately-named orig tarball in the same directory as the package-debian dir, e.g., <tt>modules/packages/manifests/mozilla</tt>.  Then run <tt>puppetagain-build-deb</tt> with the path to the package-debian dir:
+    puppetagain-build-deb precise amd64 modules/packages/manifests/nodejs-debian
+This will build the package using cowbuilder and tell you the location of the results at the end.  You'll probably need to build for both amd64 and i386.  And in the future, you may need different distributions than just "precise".
+= Modifying =
+To modify an existing package, do the same, but make the necessary changes in the package-debian directory beforehand.  Note that you will need to update package-debian/changelog to update the version number.
+To modify an existing package that is not already part of PuppetAgain, copy its debian/ directory into the source tree, make any necessary modifications, and build it.
+{{note|Do not check in the orig tarball, but do be sure to copy the orig tarball into the data tree along with the .deb packages!)}}
+== Dependencies ==
+Note that the chroot environment only pulls dependencies from the active PuppetAgain repositories.  If you are building a chain of dependent packages, you'll need to either adjust the chroot to include a temporary directory, or add each dependent package to the repositories before proceeding.
+= Creating =
+To create a brand-new package with no base package, it's probably easiest to begin with a "normal" debian packaging process, and copy the resulting debian/ directory into the PuppetAgain source tree.  Then proceed as above.
+= Landing =
+See [[ReleaseEngineering/PuppetAgain/Packages#Ubuntu: Landing Custom Repository Changes]].
+= Other Approaches =
+In addition to the instructions below, these bugs also have package building pointers relevant to their specific situation:
+* {{Bug|975034}} - Patch our current version of Mesa (8.0.2) for our EC2 test machines
+* {{Bug|975216}} - Please update libxcb on the linux test slaves to a more recent release
+* {{Bug|961279}} - [https://bugzilla.mozilla.org/show_bug.cgi?id=961279#c31 Comment 31] contains detailed instructions about how deb packages for hg 3.1.2 have been built.
+You can read how to generate a "debian" directory with what you need for "puppetagain-build-deb" by [https://bugzilla.mozilla.org/show_bug.cgi?id=975034#c4 reading this comment by rail].
+You can also read this blog post to understand the whole picture:
+[http://armenzg.blogspot.ca/2014/03/debian-packaging-and-deployment-in.html http://armenzg.blogspot.ca/2014/03/debian-packaging-and-deployment-in.html]
+== Building ==
+You can use Ubuntu 12.04 based VM on your laptop or ubuntu64packager1.
 To build a package in clean-room environment use <tt>pbuilder</tt>. It uses precreated images to build DEBs in chroot.
-== Create pbuilder images ==
+=== Create pbuilder images ===
-Use the following command to create environments.
+Use the following command to create pbuilder environments.
-* 64 bit
+* Ubuntu 12.04 Precise 64 bit
 <pre>
 sudo pbuilder --create --distribution precise --architecture amd64 \
      --components "main restricted universe multiverse" \
+    --override-config \
      --debootstrapopts --variant=buildd \
      --basetgz /var/cache/pbuilder/base-precise-amd64.tgz \
      --mirror http://archive.ubuntu.com/ubuntu \
-     --othermirror "deb http://archive.ubuntu.com/ubuntu precise-updates main restricted universe multiverse" \
+     --othermirror "deb http://archive.ubuntu.com/ubuntu precise-updates main restricted universe multiverse|deb http://archive.ubuntu.com/ubuntu precise-security main restricted universe multiverse"
-    --othermirror "deb http://archive.ubuntu.com/ubuntu precise-security main restricted universe multiverse"
 </pre>
-* 32 bit
+* Ubuntu 12.04 Precise 32 bit
 <pre>
 sudo pbuilder --create --distribution precise --architecture i386 \
      --components "main restricted universe multiverse" \
+    --override-config \
      --debootstrapopts --variant=buildd \
      --basetgz /var/cache/pbuilder/base-precise-i386.tgz \
      --mirror http://archive.ubuntu.com/ubuntu \
-     --othermirror "deb http://archive.ubuntu.com/ubuntu precise-updates main restricted universe multiverse" \
+     --othermirror "deb http://archive.ubuntu.com/ubuntu precise-updates main restricted universe multiverse|deb http://archive.ubuntu.com/ubuntu precise-security main restricted universe multiverse"
-    --othermirror "deb http://archive.ubuntu.com/ubuntu precise-security main restricted universe multiverse"
+</pre>
+* Ubuntu 14.04 Trusty 32 bit
+<pre>
+sudo pbuilder --create --distribution trusty --architecture i386 \
+    --components "main restricted universe multiverse" \
+    --override-config \
+    --debootstrapopts --variant=buildd \
+    --basetgz /var/cache/pbuilder/base-trusty-i386.tgz \
+    --mirror http://archive.ubuntu.com/ubuntu \
+    --othermirror "deb http://archive.ubuntu.com/ubuntu trusty-updates main restricted universe multiverse|deb http://archive.ubuntu.com/ubuntu trusty-security main restricted universe multiverse"
+</pre>
+* Ubuntu 14.04 Trusty 64 bit
+<pre>
+sudo pbuilder --create --distribution trusty --architecture amd64 \
+    --components "main restricted universe multiverse" \
+    --override-config \
+    --debootstrapopts --variant=buildd \
+    --basetgz /var/cache/pbuilder/base-trusty-amd64.tgz \
+    --mirror http://archive.ubuntu.com/ubuntu \
+    --othermirror "deb http://archive.ubuntu.com/ubuntu trusty-updates main restricted universe multiverse|deb http://archive.ubuntu.com/ubuntu trusty-security main restricted universe multiverse"
 </pre>
-== Building a package ==
+=== Building a package ===
 There are some different ways to build a binary package
-=== Building from orig.tar.gz, debian.tar.gz and dsc ===
+==== Building from orig.tar.gz, debian.tar.gz and dsc ====
-  sudo pbuilder --build --distribution precise --architecture amd64 --basetgz /var/cache/pbuilder/base-precise-amd64.tgz\
+  sudo pbuilder --build --distribution precise --architecture amd64 \
-     --buildresult ./out *.dsc
+ --basetgz /var/cache/pbuilder/base-precise-amd64.tgz --buildresult ./out *.dsc
 "out" directory will be populated with files you'll need to import into the repo.
 By default if the same upstream version has 2 entries in debian/changelog, packaging tools don't add orig.tar.gz in the *.changes files (used when you import the package). If for some reason you need to add the upstream source into the "changes" file (for example, when you backport some package) pass "-sa" to the packaging tools:
-  sudo pbuilder --build --distribution precise --architecture amd64 --basetgz /var/cache/pbuilder/base-precise-amd64.tgz \
+  sudo pbuilder --build --distribution precise --architecture amd64 \
-    --buildresult out --debbuildopts "-sa" *.dsc
+ --basetgz /var/cache/pbuilder/base-precise-amd64.tgz --buildresult out --debbuildopts "-sa" *.dsc
-=== Building using git-buildpackage ===
+==== Building using git-buildpackage ====
 [http://honk.sigxcpu.org/projects/git-buildpackage/manual-html/gbp.html git-buildpackage] allows you to have everything related to a package under version control. Your git checkout contains the following branches needed to build the package:
 * upstream: upstream sources
@@ Line 74: / Line 144: @@
 dch --local mozilla --distribution precise # bump debian/changelog with proper version suffix
 git commit -va
-# biuld it
+# build it
 ARCH=amd64 BUILDER=pbuilder git-buildpackage --git-upstream-branch=origin/upstream --git-debian-branch=mozilla  # add -sa to include source)
 # once you're happy with the package, tag it
 ARCH=amd64 BUILDER=pbuilder git-buildpackage --git-upstream-branch=origin/upstream --git-debian-branch=mozilla --git-tag
+# to build i386 specific package you need to pass -B to avoid adding orig/debian.tar/gz files into the changes file
+ARCH=i386 BUILDER=pbuilder git-buildpackage --git-upstream-branch=origin/upstream --git-debian-branch=mozilla  -B
 </pre>
-= Testing =
+==== Building multiple dependent packages ====
-At the moment there is no testing repo. You need to manually install packages to test them.
+Sometimes you may need to build a package which requires a fresher dependency (library). In this case you need to build the library first and use a temporary local repo. Packaging nodejs is a good example.
-= Landing =
+Pbuilder can use hooks to run some actions.
+* Add the following entries to your ~/.pbuilderrc
+ HOOKDIR="$HOME/.pbuilder-hooks"
+ OTHERMIRROR="deb file://$HOME/debs/packages ./"
+ BINDMOUNTS="$HOME/debs/packages"
+ EXTRAPACKAGES="apt-utils"
-NOTE: be careful with this not-yet-ready-to-production version.
+* Add a hook to generate package indexes
+ mkdir ~/.pbuilder-hooks
+* .pbuilder-hooks/D05deps contents:
+ #!/bin/sh
+ (cd /home/rail/debs/packages && apt-ftparchive packages . > Packages)
+ apt-get update
+* chmod 755 .pbuilder-hooks/D05deps
-We use [http://mirrorer.alioth.debian.org/ reprepro] to manage packages.
+* edit existing images
+ sudo pbuilder --login --basetgz /var/cache/pbuilder/base-precise-amd64.tgz --save-after-login
+ # inside the subshell
+ apt-get install apt-utils
+ echo "deb file:///home/rail/debs/packages ./" >> /etc/apt/sources.list
+ echo 'APT::Get::AllowUnauthenticated "true";' > /etc/apt/apt.conf.d/90mozilla
+* repeat for the i386 image
-ATM, the repo lives in /home/raliiev/repo on releng-puppet1.srv.releng.scl3.
+* build the library
+* copy the resulting debs into $HOME/debs/packages
+* build the application
-There are 2 important files under <tt>conf</tt> directory:
+== Testing ==
-* options
+At the moment there is no testing repo. You need to manually install packages to test them.
-<pre>
-verbose
-outdir +b/releng
-logdir +b/logs
-dbdir +b/db
-morguedir +b/morgue
-keepunreferencedfiles
-keepunusednewfiles
-</pre>
-* distributions
-<pre>
-Origin: mozilla
-Label: Mozilla
-Codename: precise
-Version: 12.04
-Architectures: amd64 i386 source
-Components: main
-Description: Releng repos
-DebIndices: Packages Release . .bz2
-</pre>
-Until we have a shared location (an Ubuntu machine) for package building and repo updates (using incoming directory) you need to sync that directory to your machine.
-Example import:
-<pre>
-cd repo
-reprepro -V --basedir . include precise ~/debs/puppet/build-area/puppet_2.7.17-1mozilla1_amd64.changes
-</pre>
-It will copy packages to releng/, generate indices and update the database. Make sure to not get into a race condition with other people.
-When you are done with the repo rsync it back, then publish:
- sudo chown puppetagainsync:puppetagainsync -R releng
- sudo rsync -av ./releng/ /data/repos/apt/releng/