MozillaWiki

ReleaseEngineering/How To/Process release email: Difference between revisions

Page Discussion

ReleaseEngineering/How To/Process release email (view source)

Revision as of 17:27, 2 June 2020

3,793 bytes added ,  2 June 2020
adhoc, no more cot gpg
(added link, fixed formatting)
(adhoc, no more cot gpg)
 
(23 intermediate revisions by 8 users not shown)
Line 1: Line 1:
{{Release Engineering How To|Process email to release@}}
{{Release Engineering How To|Process email to release@}}
This is a list of automatically generated emails you should expect to receive as a release engineer at mozilla.  It is not complete
This is a list of automatically generated emails you should expect to receive as a release engineer at mozilla.  It is not complete.
 
Note that email is not a good notification methodology, and better systems should always be preferred. However, it often is all that is available for audience which needs the notification. To minimize the pain of email notifications, follow these guidelines:
* email should go to a unique address for the service. This can be achieved by using "plus addresses" (preferred due to positive filtering criterea). (Note: AWS SES refers to these as "labels".)
** if not possible, the message MUST have a unique start to the subject field (brittle).
* email should be documented on this page.
 
Some email is also routed to archives, which you may prefer to search instead of joining a list to receive emails:
* [https://groups.google.com/a/mozilla.com/forum/#!forum/releng-puppet-mail releng puppet email]
* [https://groups.google.com/a/mozilla.com/forum/?hl=en#!forum/release-automation-notifications release automation email]
* [https://groups.google.com/a/mozilla.com/forum/?hl=en#!forum/releng-ops-trial vcs-sync operations email]


= Index =
= Index =
Zimbra glob/wildcard syntax, in alpha order by field then match string
''Note:'' The "Wildcard" column gives a suggestion on how to filter for that email.


{| cellpadding="10" cellspacing="0" border="1" class="fullwidth-table sortable"
{| cellpadding="10" cellspacing="0" border="1" class="fullwidth-table sortable"
Line 9: Line 19:
|-
|-
|Subject  || collapse report || [[#Performance Metrics]]
|Subject  || collapse report || [[#Performance Metrics]]
|-
|Subject  || Humpty Dumpty Error * || [[#Puppet failing too many times on a slave]]
|-
|Subject  || idle kittens report || [[#briar-patch idle kittens reporting]]
|-
|Subject ||[puppet-monitoring]* ||[[#Puppet Log Monitoring]]
|-
|-
|Subject || Suspected machine issue (* || Not an actionable email at this point. (from: nobody@cruncher - s/a {{bug|825625}}
|Subject || Suspected machine issue (* || Not an actionable email at this point. (from: nobody@cruncher - s/a {{bug|825625}}
Line 23: Line 27:
|-
|-
|Subject ||[vcs2vcs] alert_major_errors* || '''major processing error''' make sure build duty and/or hwine know [[#vcs2vcs|details]]
|Subject ||[vcs2vcs] alert_major_errors* || '''major processing error''' make sure build duty and/or hwine know [[#vcs2vcs|details]]
|-
|Subject ||[vcs2vcs]: git.m.o push N failed: * || single occurrence related to git.m.o/releases/gecko.git '''if repeated, this is a major processing error''' make sure build duty and/or hwine know [[#vcs2vcs|details]]
|-
|-
|Subject ||[vcs2vcs] process delays* || '''if repeated, this is a major processing error''' make sure build duty and/or hwine know [[#vcs2vcs|details]]
|Subject ||[vcs2vcs] process delays* || '''if repeated, this is a major processing error''' make sure build duty and/or hwine know [[#vcs2vcs|details]]
|-
|-
|Subject || [release-runner] failed || [[#What_to_do_when_one_is_received_6| How to investigate release runner failures]]. ''(ignore if a release isn't actively being started)''
|Subject || [release-runner] failed || [[#What_to_do_when_one_is_received_6| How to investigate release runner failures]]. ''(ignore if a release isn't actively being started)''
|-
|To      || release+amp@mozilla.com || Test Google Play store account. Blassey and Snorp have access. RelEng does not have access at this time.
|-
|-
|To      || release+aws@mozilla.com || AWS admin email, service notifications & marketing. See [[#Amazon_EC2_Instance_scheduled_for_retirement|list of AWS emails]], contact catlee if unsure how to handle.
|To      || release+aws@mozilla.com || AWS admin email, service notifications & marketing. See [[#Amazon_EC2_Instance_scheduled_for_retirement|list of AWS emails]], contact catlee if unsure how to handle.
|-
|-
|To      || release+bitbucket@mozilla.com || Mozilla Bitbucket Admin email (contact hwine for now)
|To      || release+bitbucket@mozilla.com || Mozilla Bitbucket Admin email (contact hwine for now)
|-
|To      || release+vcs2vcs@mozilla.com || Output from vcs2vcs hg<->git conversion ([[#vcs2vcs|details]])
|-
|-
|To      || release+aws-sanity-check@mozilla.com || Output from cruncher [http://hg.mozilla.org/build/cloud-tools/file/default/aws/aws_sanity_checker.py aws_sanity_checker.py] (contact rail)
|To      || release+aws-sanity-check@mozilla.com || Output from cruncher [http://hg.mozilla.org/build/cloud-tools/file/default/aws/aws_sanity_checker.py aws_sanity_checker.py] (contact rail)
Line 38: Line 44:
|To      || release+ec2.*@mozilla.com || error output from crontab on the indicated machine. FIX ISSUE!
|To      || release+ec2.*@mozilla.com || error output from crontab on the indicated machine. FIX ISSUE!
|-
|-
|To      || release+update.b2g.o@mozilla.com || low disk space on dogfood update server, see {{bug|877224}}
|To      || release+sns@mozilla.com || SNS issue notifications from various services. FIX ISSUE!
|-
|To      || release+chromecast@mozilla.com || Developer account for Chromecast app support {{bug|1037018}} ([[#chromecast|details]])
|-
|To      || release+v2v-gh@mozilla.com || Primary email for github account moz-v2v-gh. Contact vcs-sync folk
|-
|To      || release+roku@mozilla.com || Primary email for Roku account, mfinkle is dev contact
|-
|To      ||  release+signaddons@mozilla.com || Primary email for signing addons in automation via API
|-
|To      ||  release+ubuntu-store@mozilla.com || Primary email for Ubuntu Store
|-
|To      || release+mozdef@mozilla.com || Security alerts from infosec's Mozdef server.  Alert team&infosec if you find suspicious activity.
|-
|To      || release+moc_notifications@mozilla.com || Something from the MOC. Action depends on content. (Cited in [https://mana.mozilla.org/wiki/display/MOC/MOC+Contact+Groups#MOCContactGroups-Teams mana].)
|-
|To      || release+appleagent@mozilla.com || Related to Apple ID account -- bring to manager's attention if lots of activity.
|-
|To      || release+wcw@mozilla.com & release+wmw@mozilla.com || Requests for Wednesday Change Window (mana link to come). CiDuty or manager should respond.
|-
|To      || release+github@mozilla.com || GitHub private repo access, esp partners + xpi.
|}
|}


Line 44: Line 70:


__TOC__
__TOC__
=briar-patch idle kittens reporting=
== Why we get them ==
Email report outlining the status of any host that has been flagged as "idle"
== What is sending them ==
A cron job that is running the kittenreaper.py task with the following parameters
  python kittenreaper.py -w 1 -e
It pulls the list of hosts to check from http://builddata.pub.build.mozilla.org/reports/slaves_needing_reboot.txt
== What to do when one is received ==
not sure yet, unless your buildduty - then you should be watching it
== Future plans ==
This will be replaced by the briar-patch dashboard
== How to best filter these emails ==
Filtering can be done by matching the subject line which will not change
=Puppet Log Monitoring=
== Why we get them ==
There are messages in the puppet master logs that indicate something is wrong with a slave or master.  Since we have no other master monitoring tools, we are defaulting to sending email.
== What is sending them ==
scl-production-puppet and soon all puppet masters have an instance of 'watch-puppet.py' running under screen as root.
The code for this script is stored [https://github.com/jhford/monitor-puppet here]
== What to do when one is received ==
* if the title contains "[puppet-monitoring][master_name] <slavename> is waiting to be signed", this is for information and requires no immediate action
* if the title contains "[puppet-monitoring][master_name] <slavename> has invalid cert", the script will try once to clean the cert before sending the email once there is a waiting signing request.  If this is successful, you'll see a matching "<slavename> is waiting to be signed" email.  The key will be automatically signed by a cronjob
== How to silence or acknowledge this alert ==
It is not currently possible to silence this email.  This script will send email each time the corresponding line pattern is seen in /var/log/messages.  This means that most likely, each time a slave tries to puppet, an email will be sent.
== Future plans ==
In the short term, we'd like to have this script monitor the puppet logs for more error conditions.  It would also make sense to monitor all puppet masters
== How to best filter these emails ==
* subject includes [puppet-monitoring]
=Puppet failing too many times on a slave=
== Why we get them ==
We have no other monitoring for slaves failing to run puppet successfully.  This became a large issue with the rev4 talos machines due to {{bug|700672}}.  We are now doing an exponential back off on these slaves with a set number of iterations.  Once the maximum number of iterations is reached, the slave will send this email then reboot.  This helps us avoid puppet master load as well as allowing the machines try to fix themselves by rebooting.
== What is sending them ==
Each machine that has these emails enabled will send the email itself when it fails to puppet the last time, and right before it reboots.
The code that sends them is unversioned, but is deployed to the slaves from
scl-production-puppet:/N/production/darwin10-i386/test/usr/local/bin/run-puppet.sh
== What to do when one is received ==
* either ignore the email or find the root of the problem and fix it. 
== How to silence or acknowledge this alert ==
This email is a temporary workaround until we get a real puppet client monitoring tool.  This email we be sent each time the maximum number of retires is reached, which is every couple hours.
== Future plans ==
Would really like to replace these emails with real puppet monitoring.
== How to best filter these emails ==
These emails are best filtered by having "Humpty Dumpty Error" in their subject.  Becuase the hostname on the slave might not be correct every/all the time, filtering on domain names might not catch all cases.


=Performance Metrics=
=Performance Metrics=
Line 131: Line 93:
= vcs2vcs System=
= vcs2vcs System=
== Why we get them ==
== Why we get them ==
These emails are the interim notification for vcs2vcs system, and indicate an error that must be addressed. The b2g project is dependent upon parts of the vcs2vcs system, as are other developers.
These emails are the interim notification for vcs2vcs system, and indicate an error that must be addressed. The b2g project is dependent upon parts of the vcs2vcs system, as are other developers and partners.


== What is sending them ==
== What is sending them ==
Line 147: Line 109:
** NOTE: you may receive an additional email after the root cause is resolved. (The alert checks on the hour for problems in the prior hour.)
** NOTE: you may receive an additional email after the root cause is resolved. (The alert checks on the hour for problems in the prior hour.)


* if the subject contains "TERMINATED", this is a major problem with the update of gecko.git (interim version). Contact (page) hwine <small>(note: this comes from the interim script "keep_clean_room_updated" running in vcs2vcs@github-sync2.dmz.scl3:/opt/vcs2vcs/b2g/wip/" as of 2012-11-10)</small>
* if the subject containes "[vcs2vcs]: git.m.o push N failed for gecko.git:", this is a (usually) transient problem with pushing gecko.git (the partner facing gecko repository) to either git.m.o or git staging. Two pushes are tried each iteration - both should succeed. Each push is numbered '1' or '2', if you see only one email report, the other already succeeded, and is ignorable. One or two sets of emails is ignorable, any more needs investigation, starting with the health of git.mozilla.org. (Note that the message is short, as this also pages hwine via sms, where brevity is nice.)


* if the subject is something else, this is likely unexpected output from a cron job. Judge the severity and escalate to hwine appropriately. File a bug to get better diagnosis of this error condition in the future.
* if the subject is something else, this is likely unexpected output from a cron job. Judge the severity and escalate to hwine appropriately. File a bug to get better diagnosis of this error condition in the future.
Line 155: Line 117:


== Future plans ==
== Future plans ==
The system will eventually be transitioned to IT for operations. Specific email will be converted to nagios alerts before then.
The system will eventually be transitioned to Developer Productivity (nee Developer Services (nee IT)) for operations. Specific email will be converted to nagios alerts before then.


== How to best filter these emails ==
== How to best filter these emails ==
Line 182: Line 144:
== How to best filter these emails ==
== How to best filter these emails ==
[release-runner] in the subject.
[release-runner] in the subject.


=Amazon EC2 Instance scheduled for retirement=
=Amazon EC2 Instance scheduled for retirement=
Line 203: Line 164:
== How to silence or acknowledge this alert ==
== How to silence or acknowledge this alert ==


== Future plans ==
== How to best filter these emails ==
Filter on the sender and subject line.
=SNS Notifications from AWS=
==Example==
Anything with the Subject "AWS Notification Message"
== Why we get them ==
We use SNS to deliver notifications about various Amazon services as well as services like Papertrail. These are generally critical alerts that we've set up and should be dealt with/investigated in a timely fashion. At the moment, only AWS Cloudwatch and Papertrail use this service, but we will likely add more in the future after we get an SNS->irc bot set up because it allows for an easy HTTP/HTTPS endpoint push that other services already integrate with.
== What is sending them ==
The Amazon SNS service notification topic "buildduty"
* arn:aws:sns:us-west-2:314336048151:buildduty
* arn:aws:sns:us-east-1:314336048151:buildduty
== What to do when one is received ==
Determine what the issue is by parsing the output. Make sure someone is working on fixing the issue (if you're not sure how, at least contact ciduty for their input/advice).
== How to silence or acknowledge this alert ==
Fix the underlying issue to stop the alert.


== Future plans ==
== Future plans ==
In the near future we intend to send SNS notifications to an irc bot instead of via email.
== How to best filter these emails ==
Ideally you should not filter them except into a high priority folder. You can filter on the Subject or the To address.
<div id="chromecast"></div>
= Mail to release+chromecast@mozilla.com =
== Why we get them ==
The mobile team is adding Chromecast support (ability to fling videos/tabs from a device to a TV). They need a persistent account not linked to a single developer who might leave the company at some point.
== What is sending them ==
These emails come from the


== What to do when one is received ==
Traffic should be light. If the email is not simply Google self-promotion, please forward it to lead mobile devs, namely :blassey and :mfinkle.
== How to silence or acknowledge this alert ==
== Future plans ==


== How to best filter these emails ==
== How to best filter these emails ==
Filter on the sender and subject line.
You can either filter on the "To:" field for "release+chromecast@mozilla.com" to catch just these emails, or filter on "From:" for "noreply@google.com" and move all mail from Google (we have multiple accounts mailing us intermittently) to a separate Google subfolder (coop).
 


<hr />
<hr />
<small>
<small>
=Security Alerts from Mozdef=
== Why we get them ==
Mozdef is an ELK stack (logging aggregator + parser) run by the infosec team.  They're consuming our Papertrail logs, at our request.
2016.09.13: We have asked them to create some preliminary alerts on ssh access to our signing infrastructure.  See https://bugzilla.mozilla.org/show_bug.cgi?id=1290261
== What is sending them ==
2016.09.13: the infosec team has a cron job finding ssh activity on the signing infrastructure, and that emails us.
== What to do when one is received ==
2016.09.13: The emails are very new.  For now, we most likely want to take a look and see what the 'normal' looks like, so we know when something out of the ordinary happens.
On suspicious email, notify the team and infosec.
== How to silence or acknowledge this alert ==
2016.10.08: These will send once an hour if there is ssh access.
== Future plans ==
2016.09.13: We may change the frequency of the emails to be more immediate, once we know the noise level.
== How to best filter these emails ==
As noted in the table above, these are sent to release+mozdef@mozilla.com
=Mail to release+moc_notifications@mozilla.com=
== Why we get them ==
Unsure when MOC will use this address.
== What is sending them ==
Humans from MOC will use this address.
== What to do when one is received ==
* Read and handle
== How to silence or acknowledge this alert ==
* depends on context
== Future plans ==
Unknown - check [https://mana.mozilla.org/wiki/display/MOC/MOC+Contact+Groups#MOCContactGroups-Teams mana] to see if anything has changed.
== How to best filter these emails ==
Filter by "to" address.
=Mail to release+appleagent@mozilla.com=
== Why we get them ==
* 2 step verification
* fall back account
== What is sending them ==
Apple when folks interact with the release Apple ID agent account.
== What to do when one is received ==
* If you generated it, claim it by reply.
* Unclaimed emails should be escalated to folks with access to release Apple ID accounts
== How to silence or acknowledge this alert ==
* depends on context
== Future plans ==
''none''
== How to best filter these emails ==
Filter by "to" address.


=Sample=
=Sample=
Line 233: Line 300:
== How to best filter these emails ==
== How to best filter these emails ==
provide insight on how to filter these emails.  Is there a distinguishing header? Is it always from a specifc host, or family of hosts?  Is there a distinctive subject?
provide insight on how to filter these emails.  Is there a distinguishing header? Is it always from a specifc host, or family of hosts?  Is there a distinctive subject?
=Mail to release+adhoc-signing@mozilla.com=
== Why we get them ==
* adhoc repo release promotion
== What is sending them ==
https://github.com/mozilla-releng/adhoc-signing relpro
== What to do when one is received ==
* Make sure we're expecting an adhoc signing operation
== How to silence or acknowledge this alert ==
* depends on context
== Future plans ==
''none''
== How to best filter these emails ==
Filter by "to" address.
</small>
</small>
Asasaki
Confirmed users
4,971

edits

Retrieved from "https://wiki.mozilla.org/Special:MobileDiff/725579...1227843"