Eliatra Suite

2023-08-31

Log Analytics, Pt.3: Sending Notifications

In this article, we will use our preconfigured channels to add Email and Slack notifications to our Watch and get alerted about high error rates in our logs.

In this series’s last article, we set up a Watch in Blocks Mode that scans our sample log data for errors. In this article, we will use our preconfigured channels to send out notifications.

Adding an Email Notification Channel

As a first step, let’s add the email channel we have set up in the first article of this series to our watch.
To do so, we head to our Watch definition and add an Email action at the bottom of the page.
OpenSearch Alerting Add Notification
Next, we choose the preconfigured “DevOps Notifications” account, give the action a name, and fill in the fields “from”, “to” and “subject”. For the time being, we will leave the “Throttle Period” at its default value. We will go into throttling and acknowledging actions in a separate article.
OpenSearch Alerting Add Email

Email Body

Let’s now take care of the actual content of our email. As you can see, you can define a text body, an HTML body or both. In order to make the email readable on all devices, you probably want to use both.
As a first step, we simply want to add the number of detected errors to our Email. You can use the Mustache template language in both the text and HTML versions of the Email to insert dynamic data. In our case, it’s the total hits of our aggregate search that we set up as Watch input.
OpenSearch Alerting Email Body

Testing it Out

Now it’s time to test everything out. We scroll up to the top of the page, set the Watch to “Active” and and click on “Update”.
 OpenSearch Alerting Set Watch Active
This will take us back to the Watch Overview page. After a short while you will see that the Watch has been executed and that the DevOps Email channel action has been triggered as well.
OpenSearch Alerting Watch Executed
Now you should have received an Email in your inbox with the Email body we defined before.
OpenSearch Alerting Email

Spicing Things Up: Adding Runtime Data

Remember that a Watch produces runtime data in JSON format. This runtime data not only contains the result of your queries, calculations, and transformations, but also some metadata. For example, the Watch name and the Watch execution time. You can see all available runtime data in the preview of a Watch execution:
OpenSearch Alerting Add Runtime Data
You can use all of this data in your notifications. So, let’s go ahead and add the Watch name and the execution time. The execution time is on top level of our JSON structure, so we can insert it as variable in the Email body.
OpenSearch Alerting Email HTML
The Watch name is accessible in the field watch.id. In preview mode, this field will always contain the value inline_watch. This will be replaced with the actual Watch name on execution.
Note that we also applied some styling and added an image for fun. The resulting email now looks like this:
OpenSearch Alerting Email

Adding a Slack Channel

We now add the Slack channel that we have set up in the first part of this series to our Watch. A Watch can have multiple notification channels. As with the Email, we scroll down to the “Actions” part of the Watch definition, and click on “Add” -> “Slack”.
Since the Slack channel is already configured, we just need to give the action a name, choose the preconfigured Slack account, and define the message body.
As with the Email channel, the message body supports Mustache template syntax and we have access to all runtime data, of course.
OpenSearch Alerting Slack Channel
When the Watch is being executed, we get a Slack notification like this:
OpenSearch Alerting Slack Message

Where to go Next

In this article, we completed our Watch setup by adding two actions: An Email notification and a Slack notification. In the next blog article from this series, we will set up throttles and ackknowledge actions.
Ready to get started?!
Let's work together to navigate your OpenSearch journey. Send us a message and talk to the team today!
Get in touch