Configuring alert notifications, no UI

You learn to use the Kubernetes configuration method of configuring alert notifications that appear in the user notification system. Configuring notifications using this method has the same effect as using the UI method. The notifications supplement alert information in charts on the Manage/Hive/Compaction Observability dashboard in Grafana.

In this task, you see how to funnel alerts into a user notification system using a Kubernetes configuration available from your Database Catalog.

In the Data Warehouse service, click Overview and go to the Environments tab.
Locate an environment that you activated for CDW, which has the default Database Catalog.
Click > Show Kubeconfig.
The Kubeconfig for environment modal is displayed.
Click Copy Kubeconfig and paste the configuration into a file, and save the file as a YML file, for example, kube_env.yml.
On the command line of the cluster, set the KUBECONFIG environment variable to the path of the YML file.
```
$ export KUBECONFIG=<the-absolute-path-to>/kube_env.yml
```

Download the alert configuration using a command to fetch the configuration into the alertmanager.yml local file.

$ kubectl \
    -n istio-system \
    get configmap \
    alertmanager \
    -o "jsonpath={.data['alertmanager\.yml']}" > alertmanager.yml

global:
  resolve_timeout: 5m
 
route:
  group_by: ['alertname', 'namespace', 'service']
  group_wait: 30s
  group_interval: 5m
  repeat_interval: 3h
  receiver: notification_system
 
receivers:
  - name: 'notification_system'
 
inhibit_rules:
  - source_match:
      severity: 'error'
    target_match:
      severity: 'warning'
    equal: ['alertid', 'namespace', 'service']

Edit the alert configuration to configure the receiver object by following instructions in the Prometheus AlertManager v.0.20.0 documentation.

For example, a simple WebHook configuration looks something like this:

global:
  resolve_timeout: 5m
 
route:
  group_by: ['alertname', 'namespace', 'service']
  group_wait: 30s
  group_interval: 5m
  repeat_interval: 3h
  receiver: notification_system
 
receivers:
  - name: 'notification_system'
    webhook_configs:
      - url: https://co-alerting.prod.mycompany.com
        send_resolved: true
 
inhibit_rules:
  - source_match:
      severity: 'error'
    target_match:
      severity: 'warning'
    equal: ['alertid', 'namespace', 'service']

For information about integration possibilities, see the receiver section.

Check that your configuration conforms to YAML syntax.

Update the related configmap.

For example:

$ kubectl \  
    -n istio-system \
    create configmap \
    alertmanager \
    --from-file=alertmanager.yml=./alertmanager.yml \
    --dry-run=client -o yaml \
 | kubectl apply -f -

The alertmanager recognizes the change and automatically updates the configuration.