Eliatra Suite

2023-07-18

Log Analytics, Pt.1: Setting up Notification Channels

This article series will walk you through an example on how to monitor log files for errors and send out notifications with Alerting Plus.

In our last article, we gave an overview of the various capabilities of Alerting Plus, our Alerting and Monitoring solution for OpenSearch. In the following parts of the series, we will implement a simple log analytics use case step-by-step, based on the OpenSearch sample data.

Our first goal is to receive an email notification whenever we detect a spike in errors in the logs. We will also set up a Slack channel to send notifications to in parallel. Later on, we will use Severity Levels to decide whether to send to Slack or Email based on the severity of the detected errors.

This article will prepare some sample log data and set up the channels. In the next part, we will set up a simple watch that checks the log files index for errors.

Importing the Sample Data

As a first step, we will import the Dashboards sample data. On the home screen, click on “Add sample data” and then choose the “Sample web logs” data:

This will import some sample weblog data and also set up a dashboard with some visualizations.

A sample document in our logs index looks roughly like this (some fields have been omitted for brevity):

{
	"agent": "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; .NET CLR 1.1.4322)",
	"bytes": 123,
	"clientip": "166.168.152.39",
	"host": "www.opensearch.org",
	"index": "opensearch_dashboards_sample_data_logs",
	"ip": "166.168.152.39",
	"machine": {
		"ram": 3221225472,
		"os": "osx"
	},
	"referer": "http://nytimes.com/success/philip-k-chapman",
	"response": 503,
	"tags": [
		"success",
		"security"
	],
	"timestamp": "2023-07-15T16:11:47.086Z",
	"url": "https://www.opensearch.org/downloads/dataprepper",
	"utc_time": "2023-07-15T16:11:47.086Z",
	...	
}

Note that the document has a field response, containing the request’s HTTP response code. We will later use this field to check for errors. We will also use the timestamp field to check for errors in a specific time period.

Add an Email Notification Channel

Alerting Plus supports a couple of different notification channels:

Slack

PagerDuty

JIRA

Webhooks

Index - to write data back to and OpenSearch Index

In our case, we want to receive a notification via email if we detect an unusual error rate in our logs. While you can set up everything via the Alerting Plus API, we will make our life a bit easier and leverage the Alerting Plus Dashboards UI.

First, select “Alerting Plus” from the Dashboards Navigation:

To set up an Email notification channel, click on “Accounts” -> “Add” -> “Email”.

On the next screen, we configure the connection details of our Email server. The minimal fields we need to fill are the hostname, port, and the credentials of the Email account:

We want to ensure that our notifications are correct before sending out actual emails. For this we enable the Simulate mode of the Email channel and also set Debug to true. Simulate means that Alerting Plus will provide information about the Email it would send, without actually sending it. This is a great way to test our setup. We can disable these options once we have set up the watch and tested everything.

Add a Slack Notification Channel

Next, let’s set up a Slack notification channel as well. First, we add a new channel to Slack and call it alerting-plus-notifications. To send notifications to this channel, we must first add a Slack App. Don’t worry, this can be done with just a couple of clicks.

From the Slack Admin UI, we add a new App, give it a name and define the Slack Workspace we want to add the App to:

Next we choose “Add features and functionality” and select “Incoming Webhook”. We activate Webhooks and click on “Add new Webhook to Workspace”.

On the next screen we select the channel we want to post to. In our case, it’s the alerting-plus-notifications channel we created before.

That’s it. After the webhook has been set up, we can copy the webhook URL from the next screen:

The URL has the format:

https://hooks.slack.com/services/XXXX/YYYY/ZZZZ

We can now copy this URL, head over to the Alerting Plus Accounts page again, and add a new Slack channel. The only required entries are the name of the channel and the Webhook URL:

That’s it for this post. In the next article we will set up a watch that checks our log data, and send out notifications to the channels we created.

Articles in this series

Log Analytics, Pt.1: Setting up Notification Channels (this article) * Log Analytics, Pt.2: Setting up a Watch in Blocks Mode

Log Analytics, Pt.3: Sending Notifications

Log Analytics, Pt.4: Implementing Escalation Levels