Auto-tools/Projects/Platform Quality/Infrastructure Migration 2015: Difference between revisions

From MozillaWiki
Jump to navigation Jump to search
(→‎Basic strategy: More meat on plan)
(→‎Machine inventory: Complete move of inventory to its own page.)
 
(7 intermediate revisions by the same user not shown)
Line 3: Line 3:
Platform QA currently has a significant infrastructure in the QA Lab in MTV2 on the second floor. We wish to move the machinery to a data center for the following reasons:
Platform QA currently has a significant infrastructure in the QA Lab in MTV2 on the second floor. We wish to move the machinery to a data center for the following reasons:


- UPS backup of power.
* UPS backup of power.
- Remote control off and on.
* Remote control off and on.
- Scaling of virtual machines.
* Scaling of virtual machines.


This is a (living) plan to accomplish this. The goals is to have most of the functionality of the systems replaced by end of Q3 2015.
This is a (living) plan to accomplish this. The goals is to have most of the functionality of the systems replaced by end of Q3 2015.
Another problem is that our ESX server is currently at or over capacity. We have too many VMs on it, and we are going to need to be able to add more as new versions of operating systems ship (Windows 10, Mac OS X 10.11 El Capitan).


= Systems in place =
= Systems in place =
Line 14: Line 16:


* Sunny Day - please see [https://intranet.mozilla.org/QA/WebRTC_Lab] for more information. This is 3 1U servers plus a USB MP3 player and a USB camera. This system will be left in the lab.
* Sunny Day - please see [https://intranet.mozilla.org/QA/WebRTC_Lab] for more information. This is 3 1U servers plus a USB MP3 player and a USB camera. This system will be left in the lab.
** However, they are behind a VPN, and need to be accessible by the machines in the data center.
** However, they are behind a VPN, and may need to be accessible by the machines in the data center.
* pf-jenkins [https://pf-jenkins.qa.mtv2.mozilla.com:8080] - This Jenkins instance is completely housed on an ESX server (platform-esx-large.qa.mtv2.mozilla.com). There are many VMs supporting this instance on this ESX server.
* pf-jenkins [https://pf-jenkins.qa.mtv2.mozilla.com:8080] - This Jenkins instance is completely housed on an ESX server (platform-esx-large.qa.mtv2.mozilla.com). There are many VMs supporting this instance on this ESX server.
* platform-fusion.qa.mtv2.mozilla.com and 10.252.73.233 (misconfigured) Mac Minis each running 4 mac VMs.
* platform-fusion.qa.mtv2.mozilla.com and 10.252.73.233 (misconfigured) Mac Minis each running 4 mac VMs.
Line 24: Line 26:
* Data center (Syd, James, Van)
* Data center (Syd, James, Van)
** ESX server moved to datacenter.
** ESX server moved to datacenter.
** Non WebRTC-virtual machines migrated to data center's ESX farm.
** Non WebRTC-virtual machines migrated to data center's ESX farm to relieve capacity of ESX server.
** Sunny day machines left in lab
** Sunny day machines left in lab.
** Mac minis moved to data center.
** Mac minis moved to data center.


Line 35: Line 37:
** QUESTION: Can mozmill master see the Sunny Day steeplechase machine?
** QUESTION: Can mozmill master see the Sunny Day steeplechase machine?
*** If not, Sunny Day will need its own Jenkins master.
*** If not, Sunny Day will need its own Jenkins master.
** Some jobs can also be run on mozmill's existing VMs.
== Staging of changes ==
# Move jenkins master to mozmill (downtime over one weekend)
# Migrate VMs to data center ESX (no appreciable downtime; a few hours-1 day per VM)
# Move ESX server to data center (1 week downtime for WebRTC tests; other tests will be sharing mozmill builders)


= Machine inventory =
= Machine inventory =


== ESX Server ==
[[Auto-tools/Projects/Platform Quality/Jenkins Machine Inventory]]
 
* Hostname: platform-esx-large.qa.mtv2.mozilla.com
* MAC address: unknown
* IP Address: 10.22.72.136
 
{| class="wikitable"
|-
! Hostname !! MAC Address !! IP Address !! Move to Data Center ESX !! Notes
|-
| hello-e2e || 00:50:56:bb:39:b0 || 10.252.73.232 || True || This machine does not have a static IP Address assignment
|-
| pf-jenkins || 00:50:56:bb:4a:8b || 10.252.73.218 || False || To be replaced in mozmill.
|-
| pfnw1-ubuntu14-32-02 || 00:50:56:bb:54:f2 || Private Network || False ||
|-
| pfnw1-ubuntu14-64-01 || 00:50:56:bb:5a:6b || Private Network || False ||
|-
| pfnw2-ubuntu14-32-01 || 00:50:56:bb:52:25 || Private Network || False ||
|-
| pfnw2-ubuntu14-64-02 || 00:50:56:bb:09:11 || Private Network || False ||
|-
| pfnw-router || 00:50:56:bb:7d:1d || 10.252.73.51 || False ||
|-
| pfnw-services || 00:50:56:bb:5e:c7 || 10.252.73.50 || False ||
|-
| pf-ubuntu14-32-01 || 00:50:56:bb:54:60 || 10.252.73.130 || False ||
|-
| pf-ubuntu14-64-01 || 00:0c:29:6c:e2:6d || 10.252.73.220 || False ||
|-
| pf-win7-32-01 || 00:50:56:bb:0c:21 || 10.252.73.131 || False ||
|-
| pf-win7-32-02 || 00:50:56:bb:7d:a8 || 10.252.73.212 || True ||
|-
| pf-win7-64-01 || 00:0c:29:1e:4d:7d || 10.252.73.132 || False ||
|-
| pf-win7-64-02 || 00:50:56:bb:59:8b || 10.252.73.213 || True ||
|-
| pf-win81-32-01 || 00:50:56:bb:68:d5 || 10.252.73.133 || False ||
|-
| pf-win81-32-02 || 00:50:56:bb:24:79 || 10.252.73.243 || True ||
|-
| pf-win81-64-01 || 00:0c:29:d9:7d:93 || 10.252.73.134 || False ||
|-
| pf-win81-64-02 || 00:0c:29:b8:ac:5b || 10.252.73.215 || True ||
|-
| pf-winxp-32-01 || 00:50:56:bb:0d:e6 || 10.252.73.234 || False ||
|-
| platform-steeplechase || 00:0c:29:09:dc:7b || 10.252.73.219 || False ||
|}
 
== platform-fusion ==
 
* Host: platform-fusion.qa.mtv2.mozilla.com
* MAC address: 0c:4d:e9:ba:af:02
* IP address:  10.252.73.136
 
VMs:
 
{| class="wikitable"
|-
! Hostname !! MAC Address !! IP Address !! Notes
|-
| pf-mac10-7-64-01 || 00:0c:29:5d:32:65 || 10.252.73.141 ||
|-
| pf-mac-10-7-64-02 || 00:0c:29:92:f9:6c || 10.252.73.208 ||
|-
| pf-mac10-8-01 || 00:0c:29:33:f8:3e || 10.252.73.140 ||
|-
| pf-mac-10-8-02 || 00:0c:29:26:4f:32 || 10.252.73.209 ||
|}
 
== platform-fusion2 ==
 
* Host: platform-fusion2.qa.mtv2.mozilla.com
* MAC address ac:87:a3:21:f3:10
* IP: 10.252.73.233 (note that this machine’s entry in the DHCP mapper has the wrong MAC address)
 
VMs:
 
{| class="wikitable"
|-
! Hostname !! MAC Address !! IP Address !! Notes
|-
| pf-mac10-9-01 || 00:0c:29:0b:2d:f8 || 10.252.73.137 ||
|-
| pf-mac-10-9-02 || 00:0c:29:be:76:fd || 10.252.73.210 ||
|-
| pf-mac10-10-01 || 00:0c:29:0e:4b:55|| 10.252.73.142 ||
|-
| pf-mac-10-10-02 || 00:0c:29:de:c9:56 || 10.252.73.211 ||
|}

Latest revision as of 21:20, 8 September 2015

Introduction

Platform QA currently has a significant infrastructure in the QA Lab in MTV2 on the second floor. We wish to move the machinery to a data center for the following reasons:

  • UPS backup of power.
  • Remote control off and on.
  • Scaling of virtual machines.

This is a (living) plan to accomplish this. The goals is to have most of the functionality of the systems replaced by end of Q3 2015.

Another problem is that our ESX server is currently at or over capacity. We have too many VMs on it, and we are going to need to be able to add more as new versions of operating systems ship (Windows 10, Mac OS X 10.11 El Capitan).

Systems in place

There are two test systems currently established:

  • Sunny Day - please see [1] for more information. This is 3 1U servers plus a USB MP3 player and a USB camera. This system will be left in the lab.
    • However, they are behind a VPN, and may need to be accessible by the machines in the data center.
  • pf-jenkins [2] - This Jenkins instance is completely housed on an ESX server (platform-esx-large.qa.mtv2.mozilla.com). There are many VMs supporting this instance on this ESX server.
  • platform-fusion.qa.mtv2.mozilla.com and 10.252.73.233 (misconfigured) Mac Minis each running 4 mac VMs.

Basic strategy

Due date: End of Q3 2015
  • Data center (Syd, James, Van)
    • ESX server moved to datacenter.
    • Non WebRTC-virtual machines migrated to data center's ESX farm to relieve capacity of ESX server.
    • Sunny day machines left in lab.
    • Mac minis moved to data center.
  • Mozmill merge (Syd, Henrik, Nils)
    • Functionality of pf-jenkins to be assumed by the mozmill jenkins instance.
      • Jobs migrated.
      • Reconfigured to use the pulse notifications for build downloads.
      • New jobs to take advantage of beta, release and ESR developed.
    • QUESTION: Can mozmill master see the Sunny Day steeplechase machine?
      • If not, Sunny Day will need its own Jenkins master.
    • Some jobs can also be run on mozmill's existing VMs.

Staging of changes

  1. Move jenkins master to mozmill (downtime over one weekend)
  2. Migrate VMs to data center ESX (no appreciable downtime; a few hours-1 day per VM)
  3. Move ESX server to data center (1 week downtime for WebRTC tests; other tests will be sharing mozmill builders)

Machine inventory

Auto-tools/Projects/Platform Quality/Jenkins Machine Inventory

Retrieved from "https://wiki.mozilla.org/index.php?title=Auto-tools/Projects/Platform_Quality/Infrastructure_Migration_2015&oldid=1094306"