Eliatra Suite

2023-07-18

Log Analytics, Pt.1: Setting up Notification Channels

This article series will walk you through an example on how to monitor log files for errors and send out notifications with Alerting Plus.

Reading time: 4 minutes
By Eliatra
In our last article, we gave an overview of the various capabilities of Alerting Plus, our Alerting and Monitoring solution for OpenSearch. In the following parts of the series, we will implement a simple log analytics use case step-by-step, based on the OpenSearch sample data.
Our first goal is to receive an email notification whenever we detect a spike in errors in the logs. We will also set up a Slack channel to send notifications to in parallel. Later on, we will use Severity Levels to decide whether to send to Slack or Email based on the severity of the detected errors.
This article will prepare some sample log data and set up the channels. In the next part, we will set up a simple watch that checks the log files index for errors.

Importing the Sample Data

As a first step, we will import the Dashboards sample data. On the home screen, click on “Add sample data” and then choose the “Sample web logs” data:
OpenSearch Alerting add sample data
OpenSearch Alerting add log sample data
This will import some sample weblog data and also set up a dashboard with some visualizations.
A sample document in our logs index looks roughly like this (some fields have been omitted for brevity):
copy
{
	"agent": "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; .NET CLR 1.1.4322)",
	"bytes": 123,
	"clientip": "166.168.152.39",
	"host": "www.opensearch.org",
	"index": "opensearch_dashboards_sample_data_logs",
	"ip": "166.168.152.39",
	"machine": {
		"ram": 3221225472,
		"os": "osx"
	},
	"referer": "http://nytimes.com/success/philip-k-chapman",
	"response": 503,
	"tags": [
		"success",
		"security"
	],
	"timestamp": "2023-07-15T16:11:47.086Z",
	"url": "https://www.opensearch.org/downloads/dataprepper",
	"utc_time": "2023-07-15T16:11:47.086Z",
	...	
}
Note that the document has a field response, containing the request’s HTTP response code. We will later use this field to check for errors. We will also use the timestamp field to check for errors in a specific time period.

Add an Email Notification Channel

Alerting Plus supports a couple of different notification channels:
  • Email
  • Slack
  • PagerDuty
  • JIRA
  • Webhooks
  • Index - to write data back to and OpenSearch Index
In our case, we want to receive a notification via email if we detect an unusual error rate in our logs. While you can set up everything via the Alerting Plus API, we will make our life a bit easier and leverage the Alerting Plus Dashboards UI.
First, select “Alerting Plus” from the Dashboards Navigation:
OpenSearch Alerting  navigation
To set up an Email notification channel, click on “Accounts” -> “Add” -> “Email”.
OpenSearch Alerting add email account
On the next screen, we configure the connection details of our Email server. The minimal fields we need to fill are the hostname, port, and the credentials of the Email account:
OpenSearch Alerting email channel
We want to ensure that our notifications are correct before sending out actual emails. For this we enable the Simulate mode of the Email channel and also set Debug to true. Simulate means that Alerting Plus will provide information about the Email it would send, without actually sending it. This is a great way to test our setup. We can disable these options once we have set up the watch and tested everything.
OpenSearch Alerting email defaults

Add a Slack Notification Channel

Next, let’s set up a Slack notification channel as well. First, we add a new channel to Slack and call it alerting-plus-notifications. To send notifications to this channel, we must first add a Slack App. Don’t worry, this can be done with just a couple of clicks.
From the Slack Admin UI, we add a new App, give it a name and define the Slack Workspace we want to add the App to:
OpenSearch Alerting create slack app
Next we choose “Add features and functionality” and select “Incoming Webhook”. We activate Webhooks and click on “Add new Webhook to Workspace”.
OpenSearch Alerting slack webhooks
OpenSearch Alerting incoming webhook
On the next screen we select the channel we want to post to. In our case, it’s the alerting-plus-notifications channel we created before.
OpenSearch Alerting webhook channel
That’s it. After the webhook has been set up, we can copy the webhook URL from the next screen:
OpenSearch Alerting webhook url
The URL has the format:
copy
https://hooks.slack.com/services/XXXX/YYYY/ZZZZ
We can now copy this URL, head over to the Alerting Plus Accounts page again, and add a new Slack channel. The only required entries are the name of the channel and the Webhook URL:
OpenSearch alerting plus slack channel
That’s it for this post. In the next article we will set up a watch that checks our log data, and send out notifications to the channels we created.

Articles in this series

Ready to get started?!
Let's work together to navigate your OpenSearch journey. Send us a message and talk to the team today!
Get in touch