Configuring Notifications
With Alert Groups and Notifications, you can create groups of alerts and setup notification targets for each group. This way, you can notify different parties interested in certain sets of alerts via different methods. For example, you might want your Hadoop Operations team to receive all alerts via EMAIL, regardless of status. And at the same time, have your System Administration team receive all RPC and CPU related alerts that are Critical only via SNMP. To achieve this scenario, you would have an Alert Notification that handles Email for all alert groups for all severity levels, and you would have a different Alert Notification group that handles SNMP on critical severity for an Alert Group that contains the RPC and CPU alerts.
Ambari defines a set of default Alert Groups for each service installed in the cluster. For example, you will see a group for HDFS Default. These groups cannot be deleted and the alerts in these groups are not modifiable. If you choose not to use these groups, just do not set a notification target for them.
Creating or Editing Notifications
Browse to the Alerts section in Ambari Web.
Under the Actions menu, click Manage Notifications.
The list of existing notifications is shown.
Click + to “Create new Alert Notification”. The Create Alert Notification dialog is displayed.
Enter the notification name, select the groups to which the notification should be assigned (all or a specific set), select the Severity levels that this notification responds to, include a description, and choose the method for notification (EMAIL or SNMP).
For EMAIL: provide information about your SMTP infrastructure such as SMTP Server, Port, To/From address and if authentication is required to relay messages through the server. You can add custom properties to the SMTP configuration based on the Javamail SMTP options.
Parameter
Description
Email To A comma-separated list of one or more email addresses to send the alert email. SMTP Server The FQDN or IP address of the SMTP server to use to relay the alert email. SMTP Port
The SMTP port on the SMTP Server. Email From A single email address to be the “from” alert email. Use Authentication
Check if your SMTP Server requires authentication in order to relay messages. Be sure to also provide the username and password credentials. For SNMP: select the SNMP version, Community, Host, and Port where the SNMP trap should be sent. Also, the OID parameter must be configured properly for SNMP trap context. If no custom, or enterprise-specific OID will be used, we recommend the following:
Parameter
Description
OID 1.3.6.1.4.1.18060.16.1.1 Hosts A comma-separated list of one or more Host FQDNs of where to send the trap. Port The port where snmptrapd is listening on the Hosts.
After completing the notification, click Save.
Creating or Editing Alert Groups
Browse to the Alerts section in Ambari Web.
From the Actions menu, choose Manage Alert Groups
The list of existing groups (default and custom) is shown.
Choose + to “Create Alert Group”. Enter the Group a name and click Save.
By clicking on the custom group in the list, you can add or delete alert definitions from this group, and change the notification targets for the group.
Dispatching Notifications
When an alert is enabled and the alert status changes (for example, from OK to CRITICAL or CRITICAL to OK), Ambari will send a notification (depending on how the user has configured notifications).
For EMAIL notifications: Ambari will send an email digest that includes all alert status changes. For example: if two alerts go CRITICAL, Ambari sends one email that says "Alert A is CRITICAL and Ambari B alert is CRITICAL". Ambari will not send another email notification until status has changed again.
For SNMP notifications: Ambari will fire an SNMP trap per alert status change. For example: if two alerts go CRITICAL, Ambari will fire two SNMP traps, one for each alert going OK -> CRITICAL. When the alert changes status from CRITICAL -> OK, another trap is sent.
Viewing Alert Status Log
In addition to dispatching alert notifications, Ambari writes alert status changes to a log on the Ambari Server host. Alert status changes will be written to the log regardless if EMAIL or SNMP notifications are configured.
On the Ambari Server host, browse to the log directory:
cd /var/log/ambari-server/View the ambari-alerts.log file.
Log entries will include the time of the status change, the alert status, the alert definition name and the response text. For example:
2015-08-10 22:47:37,120 [OK] [HARD] [STORM] (Storm Server Process) TCP OK - 0.000s response on port 8744 2015-08-11 11:06:18,479 [CRITICAL] [HARD] [AMBARI] [ambari_server_agent_heartbeat] (Ambari Agent Heartbeat) c6401.ambari.apache.org is not sending heartbeats 2015-08-11 11:08:18,481 [OK] [HARD] [AMBARI] [ambari_server_agent_heartbeat] (Ambari Agent Heartbeat) c6401.ambari.apache.org is healthy