From MozillaWiki
Jump to: navigation, search

Upcoming vacation/PTO:

  • alin - aug31-sep11
  • coop - aug 3, aug 17-28
  • kmoir - aug 3
  • otilia - aug10-21, half day on July31
  • vlad - jul31 ; aug14-aug27
  • Monday Aug 3 - Holiday in Canada




We still do not have access to update DNS in order to create an ec2 instance. From what we understand, it should take a little bit longer to solve this.

Callek will ping corey about this access

Corey has some reservations, sent summary e-mail to coop and he will review/discuss more tomorrow (Tuesday) so we're waiting at least another day.


we are now able to clone the private repo, but cannot decrypt the files using our private gpg keys. I suspect that Chris didn't manage to sign those files with our public gpg keys yet.

Callek confirmed otilia has access to deploypass.txt.gpg and password.txt.gpg

Callek noticed loanerou.txt.gpg should be encrypted for otilia and vlad (not re-encrypted as of this writing, since no use without the DNS issue above being solved)



1. Callek talked to Corey Shields about the rights to update DNS. Corey replied that they do not have granular access to grant such rights, but proposed two solutions:

replacing dns pointers with IP for puppet certificates

make a block of pointers to route53 where softvision can then have control of making changes ()

Corey also noted that the second one would be more viable. arr and coop to discuss how to move forward. (Not possible to allocate block of addresses). In the mean time you can ask in #moc to add dns entries or Kim can do it for you


uploaded all the packages mentioned by Julien to the internal pypi on the relengwebadm machine

3. tried several times to re-image machine talos-linux64-ix-003, but failed

noticed that it hangs out for a long time during first boot up

managed to connect to the machine and opened puppetize.log:

"Running puppet agent against server 'puppet' Puppet run failed; re-trying after 10m"

when rebooting the instance, it will ask for kickstart credentials


  • we used the password from deploypass.txt as the puppet deployment password
  • we currently do not have access to gpg encoded files such as "kickstart.txt.gpg" (useful for debugging during re-imaging process) and "loanerou.txt.gpg" (for loaning windows machines).

So it looks like talos-linux64-ix-003 is imaged as a 14.04 machine, not 12.04 as it should be. This is why puppet is failing (I actually did this a while ago myself). So in this document, you need to ensure you pick the OS listed in step 6c Kim tried to resign gpg keys - having problems and cannot update them - getting invalid keys when trying to sign. Done by hwine. Thanks hwine! Alin and Vlad - you should be able to update your local private git repo and decrypt deploypass.txt.gpg, kickstart.txt.gpg, loanerou.txt.gpg and slave-passwords.txt


(we may repeat here) the status of the bug switched from RESOLVED to REOPENED because Ryan noticed that the machine was running at 1280x1024 (was 1600x1200)

checked the raw logs:

04:46:17     INFO - Running pre test command run mouse & screen adjustment script with 'c:\mozilla-build\python27\python.exe ../scripts/external_tools/
04:46:21     INFO -  Screen resolution (current): (1600, 1200)
04:46:21     INFO -  Changing the screen resolution...
04:46:21     INFO -  Screen resolution (new): (1280, 1024)
04:46:21     INFO -  Mouse position (current): (640, 512)
04:46:21     INFO -  Mouse position (new): (1010, 10)

it seems there's a python script that changes the resolution in this case

Q: we are not sure about the steps required to fix the problem here. Also, we are not able to access the machine via RDP.

  • I can connect via VNC, not RDP. (I use a mac). You connect and are prompted for a VNC password which is in slave-passwords.txt.
    • [alin] cloned the repo again but was unable to decrypt slave-passwords.txt.gpg. It seems that the file was not signed with our keys.
    • it should be signed now


1. discussed with :grenade and asked him for the temporary password to login to winadmin server. Changed the passwd 2. sent to Kim an email with the details for the new slave that i loaned for her



1. it appears that we still don't have a DNS entry for ""

received the following when running "":

2015-08-05 01:41:33,332 - INFO - Checking name conflicts for tst-linux64-ec2-kmoir2 2015-08-05 01:41:54,645 - ERROR - has no DNS entry

asked in #moc channel to check this and they suggested to file a bug, which I did:

Richard Soderberg [:atoll] said that the IP address was assigned to '' and it should be 'tst' instead

this usually happens (to me) when I accidentally follow the wrong sequence of steps on the Loan wiki page, i.e. for build instead of test.

2. retried to re-image talos-linux64-ix-003 using the 64-bit version of Ubuntu 12.04

checked puppetize.log and it seems that the process terminated successfully

also, I am able to ssh to that machine

still, I only see a black screen when looking into the console and if rebooted it will ask for a username and password for a short time, then it switches to a black screen again.

I don't think that's unexpected. However, talos-linuxXX machines that are re-imaged will invariably hit - in fact, this machine was in this state before it was loaned this time. Still need to find someone to dig into that bug.

3. managed to connect to several windows slaves via VNC.

noticed that for all of them, the resolution was 1600x1200.

there is a bug opened for this "wrong resolution" issue:


1. Re-imaged bld-lion-r5-004 using the following command : /usr/sbin/bless --netboot --server bsdp://; reboot. After 4 hours I tried to login but whout any success. There is a way to extract the re-image logs for a mac ? Created a bug ticket for this problem : 2. Checked the following bug : and tried to figured it out why the server is still idle and not jobs running on this slave 3. Checked the following bug : where ryanVM wrote that a re-image is need it but I didn't started until I will figured it out why the first re-image is not worked



1. we connected to aws-manager and used to following commands:

terminate ${host}
  • ping to "" still worked
  • we were able to connect to ""
  • several hours since that, we were again not able to connect
  • terminated the host and attempted to launch a new instance
  • instance seems to be launched correctly, ping works intermittently but we are again asked for a password when connecting as root (the same thing as yesterday)
    • ping to "" does not work at the moment
    • ping to works but some packages are lost (timed out requests)

there's a loan request bug for an AWS instance:, but we are not able to loan such machines at the moment due to not having permissions to update DNS

2015-08-06 05:34:06,001 - INFO - Sanity checking DNS entries...
2015-08-06 05:34:06,005 - INFO - Checking name conflicts for tst-linux64-ec2-kmoir2
2015-08-06 05:34:30,581 - INFO - waiting for workers
2015-08-06 05:34:30,874 - INFO - Using IP
2015-08-06 05:34:31,087 - INFO - subnet subnet-ff3542d7
2015-08-06 05:34:43,733 - INFO - instance Instance:i-638f67c8 created, waiting to come up
2015-08-06 05:35:18,262 - INFO - assimilating Instance:i-638f67c8
2015-08-06 05:35:18,529 - INFO - Using private IP
2015-08-06 05:35:28,577 - WARNING - cannot connect; instance may still be starting (i-638f67c8, - Timed out trying to connect to (tried 1 time),retrying in 1200 sec ...


"bld-lion-r5-083" still inaccessible

noticed that Amy assigned the bug to DCOps to netboot the machine



1. We still have problems to activate the mfa on AWS accounts. We are receiving the following error : arn:aws:iam::314336048151:user/ is not authorized to perform: iam:CreateVirtualMFADevice on resource: arn:aws:iam::314336048151:mfa/

    • coop to investigate MFA access with script



checked in AWS console and noticed that the instance no longer exists. Deleted DNS entries and marked the bug as resolved.

2. - loan request for an AWS instance

we do not have permissions to grant/revoke VPN access: will only show info about our user says that VPN access is not needed when he requester is in IT/RelOps.

we searched in IRC and noticed that Kannan Vijayan [:djvj] is not present on #it or #releng channels, just on #developers, so it is likely that he needs VPN access.

asked on IRC and Callek said that we can assume that everyone needs the access, as adding the VPN group will not hurt.

Q: is there a way to find out the groups/permissions for a requester? Kim will ping arr and ask about VPN access, then open bug In the interim, I can update your access.

3. still got some issues with the EC2 instance

the launch process goes fine and then the puppet script will run (takes some time)

when it finishes, we are able to connect as root, but the instance will shut down itself after several minutes

even if we start it again from the AWS console, it will still shut down pretty soon

tried to launch the instance using another subnet, but the behaviour is the same

if we look at the logs from the AWS console:

collectdmon[880]: Info: collectd terminated with exit status 0
collectdmon[880]: Info: shutting down collectdmon �[74G[ OK ]
* Asking all remaining processes to terminate...       �[80G �[74G[ OK ]
* All processes ended within 4 seconds....       �[80G �[74G[ OK ]
* Deconfiguring network interfaces...       �[80G �[74G[ OK ]
* Deactivating swap...       �[80G �[74G[ OK ]
umount: /run/lock: not mounted
mount: / is busy
* Will now halt
[71672119.320942] System halted.
  • runner needs to be disabled

    • coop to find bug to disable runner


Kannan no longer needs the machine

should we re-image it and then enable it in slavealloc?