From MozillaWiki
< IT
Jump to: navigation, search

Weekly updates from Mozilla IT on things we're doing to support and empower all of Mozilla in our mission keep the web a global public resource, open and accessible to all. An Internet that truly puts people first, where individuals can shape their own experience and are empowered, safe and independent.


  • THE FINAL COUNTDOWN- 2 DAYS to the SCL3 DATACENTER SHUTDOWN! Many teams across Mozilla have been working for months to migrate services out of the SCL3 datacenter that we will be closing THIS WEEK. We are down to the LAST 2 DAYS in the datacenter, with THIS WEDNESDAY, SEPTEMBER 12th being the last day that power and connectivity will be provided. Beyond that date we will be tearing down the physical infrastructure in that space. If there are any concerns please get in touch with Corey Shields.

In the spirit of Florence and the Machine:

Shake it out, Shake it out, Shake it out, Shake it out, Whoa!/ And it's hard to dance with a server on your back so shake it out/ 'Cause we are done with SCL3/ So Wednesday we'll cut it and we'll be free/ Cause we like to keep our apps running strong/ Cloud and MDC are our new dawn!


  • THE FINAL COUNTDOWN- 16 DAYS to the SCL3 Datacenter shutdown! Many teams across Mozilla have been working for months to migrate services out of the SCL3 datacenter that we will be closing soon. We are down to the final days in the datacenter, with September 12th being the last day that power and connectivity will be provided. Beyond that date we will be tearing down the physical infrastructure in that space. If there are any concerns please get in touch with Corey Shields. IT'S THE FINAL COUNTDOWN!
  • VPN Move - As part of The Final Countdown, the Mozilla VPN is moving out of SCL3. Reiterating the email that was sent on Friday: If you use the VPN, you must download a new configuration bundle at and use that. The old VPN will be turned off on September 12th. Any transition issues, please bring up with #servicedesk on Slack.
  • AirMozilla public access unavailable until September 22, 2018. Any public events will need to be viewed on the Mozilla YouTube Channel or by logging in with any email address. Once a security update has been moved to production on September 22, will again point to the public page so that public webcasts are easily available via a single click. Until then, will point to Public webcasts are always available on the Mozilla YouTube Channel or by logging in to airmo with any email. Staff and NDA Mozillians will need to log in via SSO to view secure content. Thank you for watching AirMozilla!
  • Mozilla Tests and Experiments- With lots of exciting tests and experiments happening across Mozilla, it can sometimes be hard to know where you can find all of the tests. IT has put together a quick reference page on Mana so you can easily find those tests you want to contribute to. We will also refer new Mozillians to this page during new hire on-boarding going forward.


  • We've receieved an initial order of new MacBook Pros! In case you haven't heard, Apple officially announced the release of a new line of MacBook Pros. We are happy to announce that with the support of the IT VMO team an initial order has been received. The Client System Engineering, AVOps, and NetOps teams have completed validation on each of the new models. The 2018 Macbook Pros have been added to the Hub and we are ready to start taking your orders. Due to limited stock availability and current inventory we will make the 2018 models available only for refreshes of current systems. They will not be available to new hire until we have gone through our current stock of 2017 models
  • THE FINAL COUNTDOWN- 23 DAYS to the SCL3 Datacenter shutdown! Many teams across Mozilla have been working for months to migrate services out of the SCL3 datacenter that we will be closing soon. We are down to the final days in the datacenter, with September 12th being the last day that power and connectivity will be provided. Beyond that date we will be tearing down the physical infrastructure in that space. If there are any concerns please get in touch with Corey Shields. IT'S THE FINAL COUNTDOWN!


  • Let’s train databot! Many thanks to those who joined the first session of "Let's train databot”. We got so much interest and so much positive feedback from the first round, and we are making immediate improvements in response. The second focus group session to test the @databot Slackbot has been rescheduled to this Wednesday, August 15th at 7:30-8:30AM. Please join us so that we can train the improved version of the databot. Meeting was re-scheduled and updates sent by Josephine Tanumijaya (Slack @josephine)
  • IT Brown Bag- August 2018- Mozilla's Community Participation Guidelines. Join the IT Brown Bag in Vidyo #BrownBags channel or on AirMo this Friday, 8/17, at 12 noon PDT. Kate Mancuso will be leading us through a session on Mozilla's Community Participation Guidelines. The CPG's are a great tool for creating inclusive communities and for fostering consensus and cooperation. Join us Friday to learn how to apply them in your work to make the Web a welcoming place for all.
  • Get a little education! Come join my learning experiment in the #little-learnings Slack channel . Every day or so, get a little lesson on some topic, never more than 3 minutes long. Currently we're in the middle of Mozilla's Journey From The Datacenter to the Cloud, and after that journey wraps up next week, we'll have something new!
  • THE FINAL COUNTDOWN to the SCL3 Datacenter shutdown! Many teams across Mozilla have been working for months to migrate services out of the SCL3 datacenter that we will be closing soon. We are down to the final month in the datacenter, with September 12th being the last day that power and connectivity will be provided. Beyond that date we will be tearing down the physical infrastructure in that space. If there are any concerns please get in touch with Corey Shields. IT'S THE FINAL COUNTDOWN!


  • Fancy a little learning? Check out #little-learnings on Slack!' As Enterprise Solutions Architect, part of my role at Mozilla is to enable all Mozillians to envision routes to the future. Reaching the future requires new skills and ways of thinking. I've set up the #little-learnings channel as an educational service and experiment. Just about every day, I'll post a single slide or map, and give a little context in the Slack channel. It's a little 2-3 minute learning morsel for you. Sometimes technical, sometimes philosophical, but always as interesting as I can make it. I hope you'll join me and let me know if this is valuable. Thank you!


  • Vidyo migrated to new data centers! After 8 intense hours this past Saturday, AVOPS has successfully migrated Vidyo out of our SCL3 datacenter. All of Mozilla’s Vidyo conferences are now load balanced across MDC1 and MDC2, to enable better geolocation for our Vidyo conferences across the planet. Additionally, this is one step closer to a VidyoWebRTC solution for our environment. There was an immense amount of effort put into this task from all across the IT org. If you have any questions about Vidyo, please contact the team in the #avops Slack channel
  • The Vidyo migration has been a large-scale effort. Special thanks to John Bircher and Justdave from Netops, Chris Knowles from Infrastructure and Operations, and Van Le from the Data Center Operations team for all your hard work and attention to detail to make the migration a success!
  • Coming soon! More educational resources from IT! I'll be doing a short slide-a-day review of my OSCON talk Mozilla's Journey From The Datacenter to the Cloud in a Slack channel soon, as we try to find more ways to efficiently bring you the education and tools you need to power up your efforts to preserve the Open Web. Watch the Slack #announcements channel for more info later this week.


  • OSCON- Last week I attended the O'Reilly Open Source Conference here in Portland, along with several other Mozillians. It was powerfully different from past OSCONs. The crowd was diverse across visible and technological spectrums in a way I hadn't seen before. Everyone was talking about how to open up everything, and we're talking big players as well as small. Mozilla is a spiritual parent of OSCON, as was apparent every time someone saw me in a Mozilla shirt and asked about us, or talked about the origin of therm Open Source. I spoke about Mozilla's Journey From the Datacenter To The Cloud. In last Friday's IT Brown Bag, available on AirMo, I give a recap, along with some teachings from sessions I attended.
  • From our InfoSec team, April King released her long awaited TLS Cerfificate Viewer ‘certainly-something’ providing easy-to-read insight into the certificate status of sites you visit with Firefox. Install it and take it for a spin today! Thank you so much to April for your hard work and to InfoSec for all you do in keeping us safe as we empower our missions!


Andrew Krug guest hosting.

  • Two focus group sessions will be held to test @databot, a slack application that would help you to find relevant metrics by asking a question to the databot. Josephine T (JT) is inviting you to join the sessions at: Tuesday, July 17th at 10:30-11:30AM, Wednesday, July 25th at 7:30-8:30AM. Invitation details were sent to all moco mofo (at July 11th).


  • A shout-out to MDN! Over the last week, three different non-technical people here in Portland have come up to me and said Thank you Mozilla for MDN! MDN is a powerful tool for all web users in our daily work to keep the web open and independent. One of the MDN fans told me it was his professor that told all the new coders that MDN is where you need to go to learn to code the web. Thank you everyone who works on and contributes to MDN!
  • Debugging The Technical Interview. Are you near Portland and looking for some guidance about interviews? Next Monday at 5:30 PM at Mozilla Portland, we'll be hosting a workshop by Cate Connolly from about how to improve technical interviews. How we interview and who we hire has a huge impact on our engineering communities. Join us if you're in the area, and get free tickets at the link here. I'm still working out the details to see if we can broadcast and/or record the workshop.
  • Create your own events in Moderator! Moderator is an application developed by our Open Innovation/ Participation Systems team for collecting questions before or during an event. Moderator events can be set up for NDA Mozillians or public access, and posters must log in, but can post anonymously. We have been using Moderator for years for internal meetings, and it is now available for wider use. Click on the Create Event link in Moderator to create your event. Consult the thread in Discourse for more information and to ask questions about this great new feature to enable participation! Thanks to Tasos Katsoulas and the whole ParSys team for your work on Moderator.


  • DevOps Cohort This Friday, July 2nd! Join the DevOps cohort in the DevOps Vidyo channel or watch on AirMo. The DevOps cohort is a monthly meeting for all Mozillians interested in topics related to DevOps, the intersections between development and operations. The meeting is the first Friday of every month, 9 AM PST. This week, Daniel Hartnell will be sharing what we've learned about Amazon's Elastic Container Service for Kubernetes (EKS).
  • Interested in learning more about how we track contribution at Mozilla and what you can learn from this? Bitergia, the company behind our Community Analytics Dashboard, which is based upon the open source GrimoireLabs product, will be in our San Francisco office Friday, July 20th. Check out the incredibly rich Community Anaytics Dashboard, full of information about all sorts of things the Mozilla community works on, like Github PRs, Bugzilla, Stack Overflow postings, and so much more! If you'd like to participate in a tutorial or a Q&A on July 20th, reach out to Susy Struble (@susy on Slack). The Bitergia agenda is still being finalized but there will be remote participation for these events.


  • Shiny/New WPR tech! Badging and camera systems are comfy in their new homes across all offices, keeping you safe. The backend systems have moved into our new data center as well. In the process the team (WPR, EUS, NetOps, Service Desk, Systems, Security Vendor, PMO) upgraded 10 different systems and executed 50+ tests across 9 different offices. Years of tech debt paid down! Special kudos to Jen Hayashi, Ashish Vijayaram, Brian Hourigan, JustDave, Frances Reda,Lisa Gray, Diana Perez, and everyone else involved in the project!
  • Help us test the MDC Data Bot. The MDC Data Bot is a Slack-based application that enables searching the Mozilla Data Collective from within Slack. It's a great way to bring data into your discussions! Thank you Josephine Tanumijaya for your presentation and your great work with MDC and the MDC Data Bot!
  • New printers are coming to MTV, PDX and SFO. We will be refreshing the current fleet of Ricoh printers in the coming weeks. Look for more information in your inbox.
  • Audio Visual Incident form made simpler! Remember when you had to answer 20 questions to open an AV ticket on the Hub? Well...that’s a thing of the past. We always hope that your audio, video, and Air Mozilla experience works seamlessly during your meetings.That, however, is not always the case. When you do encounter issues, please visit the Hub, select Get help: Audio Visual, answer 3 simple questions and Help will be on the way!Making conferencing effortless is our mission, so, let the AV team help you.
  • IT is having a Back-to-School Hackathon! At our All Hands last week we began work on our first-ever IT Hackathon, a Back-to-School Hackathon on September 13th. We're working to develop an experimental, future-facing mindset in IT, and our first hackathon will give us a chance to exercise this new muscle. Tentative themes we'll be hacking on include conferencing, security, engagement, and service lifecycle management.


  • Keep your devices safe from theft! Device theft is on the rise, especially in urban areas. Device theft at Mozilla has also steadily risen over the years, with the current year being the worst over the past four years. To reduce risk to personal property, and Mozilla property and data, IT sent out an email today containing a list of Physical and Data safety practices you can take. We also have included instructions for what to do if your device is lost or stolen which we recommend skimming through when you get the chance. If you have any questions about the content of this email or about general Device safety please don’t hesitate to contact the Service Desk via email or on IRC/Slack
  • 1Password 7 app is here! This is a redesign of the 1Password 6 desktop app built from the ground up. 1Password 7 is available now for Mac and PC. Updates include: Faster item lookup and improved vault management. And there's new support for Windows Hello which will allow you to log in to your 1Password account using your fingerprint or simply your smile :). Available to download by navigating to the “Get the Apps” section when logged in to the 1Password web interface. For helping getting the new version setup or for any 1Password questions feel free to contact the Service Desk via email or on IRC/Slack
  • OpenVPN upgrade: An upgrade to our OpenVPN server went live on June 2nd, 2018 in order to to bring our environment in-line with current versions. For most users this will have no effect and no action will be necessary, however a select few may need to make some changes. An email was sent out by The Service Desk last week detailing the upgrade. You can contact the Service Desk via email or on IRC/Slack for questions or concerns about the upgrade.
    • Make sure you’re on the most up to date version of Viscosity
    • If on Linux, check that your VPN client is using OpenVPN v.2.3.3 or later
    • And Ubuntu 14 LTS is no longer supported at all, so make sure you update to a later version of Ubuntu LTS

When the lights go down in the city/ And the sun shines on the bay/ Oooo I want to be there in my city/ with my fellow MO-ZILL-IANS!


  • The Mozilla IT Brown Bags are held the 3rd Friday of every month, at noon Pacific time, and are open to all staff and NDA volunteers. They are broadcast live, and recordings are archived, on the IT Operations AirMo channel. Last week's session featured a wonderful presentation on Spectre and Meltdown at Mozilla by Michal Purzynski. Michal's presentation was engaging for people of all technical levels. If you're curious about the algorithms and underpinnings of these vulnerabilities, or want to know how they affect you, check out the recorded presentation on AirMo!
  • Last week, the IAM Team met in San Francisco for a UX workshop.
    • The UX workshop is the first effort to create user journeys and wireframes for the evolution of a replacement for the web properties and
    • I.A.M stands for Identity and Access Management. The IAM team is comprised of members of Open Innovation’s Participation Systems and IT’s InfoSec teams. The project started a little over a year ago. Its mission is to ensure all Mozillians (paid staff and contributors) have convenient and appropriate access to Mozilla services through a unified, authoritative, integrated identity system that empowers them to build a better Internet.
    • For more information on the effort around consolidating these two very important systems you can attend the fortnightly IAM Sprint Review. The next sprint review is on Thursday May 24 @ 8AM Pacific in the Parsys Vidyo channel. Special thanks to Henrik Mitsch, Thomas Gläser, Rina Tambo Jensen, and Megan Branson for the great research leading up to the workshop.
    • Code and docs at
The IAM team hard at work at the workshop


  • The next Mozilla IT Brown Bag session will be this Friday, May 18th, at 12 noon PDT on the IT AirMo channel and in the #brownbags Vidyo room. The topic will be Spectre and Meltdown at Mozilla, presented by Michal Purzynski. Come join us to learn about how these vulnerabilities affect Mozilla and what we’re doing to protect ourselves. More information about the IT Brown Bags here:

  • We are now live with our Firefox-Enterprise Web-To-Lead integration in our Salesforce CRM. This was a coordinated cross-org (MoFo,MoCo) and cross functional effort that we got completed in roughly 2 weeks. We worked closely with Ben Sternthal's MOZMEAO (pronounced Moz-Meow) team, Michele Warther’s Lifecycle Marketing team, and Porfirio Landeros on the FF Enterprise Product Marketing Team. Two days after launch we have already captured over 500 Leads from interested I.T. professionals across the globe with a potential install base of over 500K computers!


  • We’ve enabled the new Gmail UI as part of the Early Adopter Program! Those who want to try it can do so, in the Gmail web interface, by going to Gmail > the gear icon > "Try the new Mozilla Mail". Right away, you’ll see the UI redesign with new sidebar, snooze options and smart reply. Over the next few weeks you’ll see other new features like confidential mode (emails with expiration dates), nudging (reminders on important emails that you have yet to respond), and better security warnings.

Click on the gear to unlock the new Mail UI

  • The Gmail UI update is brought to you by our Google Admin team that quietly keeps all of our Google apps up and running. Thank you to Tristan Thomas (SF), Henry Langi (MTV), and Jen Hayashi (MTV) for keeping up the Google apps that are so important to our work!

Tristan Thomas, Henry Langi, Jen Hayashi


  • Thank you so much Andrew Krug for hosting the MinIT the last two weeks! Andrew did some great things for the MinIT and he and I will be collaborating on future MinITs! I'm so happy to share the MinIT with him, and with all of Mozilla!!!
  • I was at Coachella/ Leaning on your shoulder/ Watching your husband swing in time/ I guess I was in it/ Cause baby, for a minute/ It was Woodstock in my mind -Lana Del Rey
  • I had an incredibly inspiring 2 weeks at Coachella. Lovely people from all over the world gathered to celebrate music and art, and participate in a vision of a peaceful, colorful future for all. I'd like to share some Coachella slides with you...


  • MVK is still at Coachella this week. Andrew Krug guest hosting again.
  • Errata - Last weeks IT Brownbag on Meltdown and Spectre was replaced by an IT wide AMA Session with Denelle Dixon, recordings are available on AirMo. Stay tuned to the brown bags wiki for updates the new date for Michal’s presentation on Meltdown and Spectre. AMA Recording Link
  • Wednesday at 10am there will be an Operations All Hands as well for those who would like to attend. The session will be streamed on Air Mozilla in addition to the recording for non-PST viewers. AirMo Link
  • Today you may receive a survey regarding your conferencing experience here at Mozilla. We ask that you fill out the survey relating to Vidyo or any other conferencing solution you use, in order to help determine the direction of conferencing at Mozilla.
  • No more IRC gateway on slack: Starting on May 15th, it will no longer be possible to connect to Slack using the IRC and XMPP gateways. Slack Announcement If you're using Slack in a desktop based IRC client you'll need to switch to a slack API based alternative. If you are a WeeChat user there is a great Slack API plugin. WeeSlack
  • Phishing Exercise ( the reveal ) - Infosec ran a phishing exercise last week! For those of you who got it, thanks for playing! Emails came in the form of a fake mail from and directed users to a dummy auth0 page. InfoSec will host a session at the SFO All-Hands to share results, what we learned, and what we’re doing (or have done by then) to improve for next time. For any questions on the internal phishing assessment reach out to Jonathan Claudius. For more information on how you can protect yourself and learn to better spot phishing emails visit this great page from InfoSec. Phishing Guidance


  • MVK is at Coachella this week and next. Andrew Krug guest hosting.
  • Good news everyone! Mozilla's instance of 1Password will be getting DUO 2nd Factor Authentication coming soon to a browser near you.
  • Speaking of browsers VidyoConnect WebRTC on track to start Internal AV Beta testing by next week. If interested and you want to participate contact Tony Recendez.
  • Bugzilla is now live in Cloud Services AWS and out of our data center in Santa Clara. Congratulations Bugzilla and everyone who worked so hard to move to the Cloud.
  • Nubis v2.2 has just been released with a number of fixes and improvements for AWS users taking advantage of the platform. CHANGELOG
  • IT has kicked off a company wide service management project and is calling for participation. Let's all name the service management framework that will help us provide the best user experience! Service management frameworks are evolved from concepts like ITIL or ITSM but this particular framework will be uniquely Mozilla and align with our Open by Design Principles. You can get involved by joining the #itsm_all channel on slack or by commenting on the discourse thread linked here: Discourse Thread on the Project
  • Brownbag Inbound! InfoSec will debrief on April 20, 2018 at 12:00 PST in Brownbags Vidyo on the response to Meltdown and Spectre. Learn more about meltdown and spectre before the presentation here: MeltDown Attack Information


  • On March 31st our DNS and DHCP infrastructure within the SCL3 data center and external DNS was migrated from an old homebrew system to “Infoblox” appliances. This will provide a more robust service and pave the way for further service decommissioning as we work to exit the SCL3 data center. Big thanks to everyone involved in making this happen, including Rob Tucker, Brian Hourigan, Van Le, John Bircher, Vien Doan, and Krishna Nagaraja
  • What is DNS? DNS is the Domain Name System, a foundational piece of Internet technology. DNS is a common standard for mapping named hosts to addresses, and is fundamental to the way the Web functions. DNS as we know it today has been in service since 1985. DNS emerged from 10 years of standards debates, and it launched when servers were big and long-lived, and it has adapted to service the ephemeral Cloud-based web where hosts come and go every second. The story of DNS is core to the story of the decentralized Internet. If you'd like to learn more about DNS, here's links- The history and prehistory of DNS , A Slate article about the importance of DNS and its history, An intro to the tech of DNS,and Wikipedia on DNS.
  • Mozilla IT would like to especially recognize Rob Tucker- the pride of Stow, Ohio- for his work along with the rest of the team listed above in seamlessly transitioning office authoritative DNS (across MTV, SFO, TPE, BER) from Inventory to Infoblox with minimal impact and downtime. Additionally, I'd also like to thank Rob for coming to the rescue over and over again with his code skills for services that have needed special help. Rob's skills are the duct tape holding some of our more fragile systems together. Thanks Rob!

Rob Tucker phonebook.png

  • I'm headed to Coachella for 2 weeks to revel with my music tribes and to glimpse a possible future of love, peace, and mixed reality! Though I will be tech free... Out in the desert Friday night, when SZA sings that "All the Stars are Closer" and the Weeknd declares he’s a mofo Starboy, I’ll be reorienting myself towards a cosmic future. Andrew Krug from InfoSec will bring you the MinIT the next two weeks. I'll see you all in the stars!

MinIT Archives