Build:StageMigrationNotes - Revision history

NThomas at 11:09, 19 September 2007

2007-09-19T11:09:34Z

← Older revision		Revision as of 11:09, 19 September 2007
Line 88:		Line 88:
	\|}		\|}

			To be integrated into this process
			* Transfer home directories for contributors from surf to manna

	== Meeting Notes: 14aug2007 ==		== Meeting Notes: 14aug2007 ==
	*		*
	*		*

Joduinn: /* Process */

2007-08-14T22:28:58Z

Process

← Older revision		Revision as of 22:28, 14 August 2007
Line 87:		Line 87:
	* Verify end-to-end tinderbox -> ftp.m.o status		* Verify end-to-end tinderbox -> ftp.m.o status
	\|}		\|}


			== Meeting Notes: 14aug2007 ==
			*
			*

Preed at 19:16, 14 September 2006

2006-09-14T19:16:48Z

← Older revision		Revision as of 19:16, 14 September 2006
Line 30:		Line 30:
	* As part of this plan, we will be auditing access to manna; if access is removed, but it shouldn't have been, please email build@mozilla.org.		* As part of this plan, we will be auditing access to manna; if access is removed, but it shouldn't have been, please email build@mozilla.org.
	* Builds may be spotty on Friday; we'll do our best to ensure the builds get uploaded as expected.		* Builds may be spotty on Friday; we'll do our best to ensure the builds get uploaded as expected.

			= Process =

			This is a rough, ordered timeline of the steps necessary to fulfill the requirements.

			{\| class="fullwidth-table"
			! Step
			! Build
			! IT
			\|-
			\| Prerequ.
			\|
			* Initial list of rsync directories from surf -> netapp
			\|
			* Netapp mounts available to surf (r/w)
			* Netapp mounts available to manna (r/o)
			* LDAP up and running on both for authentication?
			* Reduce TTL on stage.m.o in anticipation of switchover
			\|-
			\| 1
			\|
			\|
			* Move surf inside firewall
			* Connect Netapp mount to surf
			\|-
			\| 2
			\|
			* rsync the contents of surf -> the netapp
			\|
			* Setup/verify as appropriate the http/rsync/ftp configs on manna; test with builds that begin to be available (it may make sense to make stage.m.o/ftp.m.o completely unavailable to mirrors/outsiders at this point, since we don't want the mirrors picking up incomplete directory tree)
			\|-
			\| 3
			\|
			* Delete the contents of the archive rsync module on surf; delete ftp logs on surf
			\|
			* Verify that ftp.m.o/stage.m.o appear as before, but accessed via manna (can diff manna's local disk vs. the Netapp to verify)
			\|-
			\| 4
			\|
			* Point Tinderboxen at surf
			\|
			* Start manna-upload-area -> surf rsync scripts on surf
			\|-
			\| 5
			\|
			* Verify builds in the build farm get to surf
			* Verify builds outside the build farm can get to manna's upload area
			\|
			* Setup virus scanning script
			* Start surf -> Netapp rsync scripts
			\|-
			\| 6
			\|
			* Verify end-to-end tinderbox -> ftp.m.o status
			\|
			* Verify end-to-end tinderbox -> ftp.m.o status
			\|}

Preed: /* Migration Plan */

2006-09-12T18:06:25Z

Migration Plan

← Older revision		Revision as of 18:06, 12 September 2006
Line 19:		Line 19:
	# surf/stage will have a read/write mount to shared, central storage.		# surf/stage will have a read/write mount to shared, central storage.
	# manna will be renamed stage (as a CNAME), and will mount this shared, central storage read-only. This shared, central storage will become the new definitive copy of ftp.m.o.		# manna will be renamed stage (as a CNAME), and will mount this shared, central storage read-only. This shared, central storage will become the new definitive copy of ftp.m.o.
	# Some tinderboxen will change to publish their builds to ~~stage~~		# Some tinderboxen will change to publish their builds to surf
	# Bits in the archive.m.o rsync module will move entirely to the shared, central storage.		# Bits in the archive.m.o rsync module will move entirely to the shared, central storage.
	# Bits in the release.m.o rsync module will continue to exist on surf and the shared central storage		# Bits in the release.m.o rsync module will continue to exist on surf and the shared central storage

Preed: /* Migration Plan */

2006-09-11T23:05:28Z

Migration Plan

← Older revision		Revision as of 23:05, 11 September 2006
Line 22:		Line 22:
	# Bits in the archive.m.o rsync module will move entirely to the shared, central storage.		# Bits in the archive.m.o rsync module will move entirely to the shared, central storage.
	# Bits in the release.m.o rsync module will continue to exist on surf and the shared central storage		# Bits in the release.m.o rsync module will continue to exist on surf and the shared central storage
			# surf will pull bits onto its local disk via a scheduled process and perform various verification steps on them (virus scanning, permissions checks, etc.); when these checks pass, they will be uploaded to the shared, central storage, and immediately appear on the ftp mirrors/ftp.m.o.

	= How These Changes Affect External Contributors =		= How These Changes Affect External Contributors =

Preed at 23:01, 11 September 2006

2006-09-11T23:01:33Z

New page

<h1>Stage Migration Notes</h1>

= Current Configuration =

Currently, contributors and Tinderbox machines upload directly to <tt>surf</tt> (CNAMEed to stage.m.o). <tt>manna</tt> (CNAMEd to ftp.m.o) pulls these builds from surf via rsync at set intervals.

<tt>surf</tt> is responsible for virus scanning and verifying other build-related issues.

Both surf and manna are constantly dangerously low on diskspace; the goals of this migration are to:

* Solve this continuing disk space problem
* Ensure that bits are virus scanned and otherwise verified before reaching the mirrors, which is currently not the case. Infected bits will eventually get removed from ftp.m.o/mirrors, but it's not instantaneous.

= Migration Plan =

On the evening of Thursday, 14 September, from 6 pm to midnight PDT:

# surf/stage will be moved inside the firewall and become responsible for virus scanning and verifying all bits that go to the mirror farm/ftp.m.o. Because of this, access will be restricted to this host.
# surf/stage will have a read/write mount to shared, central storage.
# manna will be renamed stage (as a CNAME), and will mount this shared, central storage read-only. This shared, central storage will become the new definitive copy of ftp.m.o.
# Some tinderboxen will change to publish their builds to stage
# Bits in the archive.m.o rsync module will move entirely to the shared, central storage.
# Bits in the release.m.o rsync module will continue to exist on surf and the shared central storage

= How These Changes Affect External Contributors =

* External contributors may have to accept a new SSH host key; this key will be published after the switchover is complete, so contributors can verify it before accepting it
* Access to surf will be limited
* As part of this plan, we will be auditing access to manna; if access is removed, but it shouldn't have been, please email build@mozilla.org.
* Builds may be spotty on Friday; we'll do our best to ensure the builds get uploaded as expected.