OpenSearch

2022-02-23

OpenSearch with cert-manager: Part 1 (Setup)

Using cert-manager in Kubernetes to generate (and more importantly automatically rotate) TLS certificates on OpenSearch and Dashboards.

A little while ago I wrote a blog on “Using OpenSearch on Kubernetes”, which covered how to:

Install custom certificates

Load custom Elasticsearch configuration

Load custom Kibana configuration

Load custom security configuration

Troubleshoot common issues

This blog is going to cover the next natural step in the process of running OpenSearch in Kubernetes: using cert-manager to generate (and more importantly automatically rotate) TLS certificates. Removing the need for manual intervention when it comes to managing TLS certificates and therefore reducing the chances of human error and possible downtime. As an added bonus, since this process is done automatically, you can utilize short lived certificates, thus increasing the security of your OpenSearch deployment.

We are Going to Cover:

Part 1

Deployment of cert-manager in k8s

Deployment of Ingress Controller and Resource

Part 2

Creation of self-signed CA using cert-manager (Optional)

Creation of TLS certificates signed by self-signed CA using cert-manager

Using these certificates in deployment of OpenSearch/Dashboards using Helm

The technologies we are going to use are:

Amazon Elastic Kubernetes Service (EKS) Cluster

Amazon Route53

cert-manager

Ingress NGINX

Helm

All configuration files and commands are available on GitHub: https://github.com/Anthony7774/k8s-cert-manager-opensearch

So lets begin!

The first step, of course, is to have cluster with Kubernetes running, I will be using Amazon EKS cluster, started using below command:

eksctl create cluster \
--name opensearch-cluster \
--region eu-west-1 \
--node-type t2.medium \
--nodes 3

As I am based in Ireland, eu-west-1 was chosen, select the region that makes sense to you.

N.B. Deleting cluster can be done with:

eksctl delete cluster --name opensearch-cluster

However, note that the PV remains on AWS and will accumulate charges if not deleted manually

1. Deployment of Cert-manager in k8s

Cert-manager is an open source tool donated by Jetstack in 2020 to the CNCF. It is a fantastic way to manage TLS certificates in Kubernetes cluster. The certificate renewal process is seamless and the configuration is very straightforward. Making this tool a must have for any cluster using TLS certificates. The docs are very good.

Install the cert-manager:

kubectl create namespace cert-manager
helm repo add jetstack https://charts.jetstack.io
helm repo update
helm install cert-manager jetstack/cert-manager --namespace cert-manager --version v1.7.0 --set installCRDs=true

The CRDs that are going to be installed are:

Issuer

ClusterIssuer

Certificate

CertificateRequest

Order

Challenge

If everything works as expected you will not have to deal with these CRDs individually, however these are great for debugging cert-manager’s actions behind the scenes.

Once this is installed successfully, which should only take a minute or two, below command will display the pods now running in cert-manager namespace:

kubectl get pods --namespace cert-manager

You should see 3 pods running as follows:

NAME                                       READY   STATUS    RESTARTS   AGE
cert-manager-847544bbd-dnm54               1/1     Running   0          37s
cert-manager-cainjector-5c747645bf-qh98q   1/1     Running   0          37s
cert-manager-webhook-f588b48b8-tgb6r       1/1     Running   0          37s

2. Deployment of Ingress Controller and Ingress Resource

As this is a new cluster we do not have any ingress controller and therefore no ingressClass available.

This can be verified by running the below command:

kubectl get ingressClass

You should see “No resources found”.

To install Ingress Controller, first create a yaml file requesting network-lb nlb instead of classic lb (which is default in Amazon cluster):

---
controller:
  service:
    annotations:
      service.beta.kubernetes.io/aws-load-balancer-type: nlb

Then run below command:

helm install ing ingress-nginx/ingress-nginx \
  --namespace ingress \
  --version 4.0.1 \
  --values ingress_controller.yaml \
  --create-namespace

Once this is installed, you can run the below command again to view the ingressClass created:

kubectl get ingressClass

You should see something like this:

NAME    CONTROLLER             PARAMETERS   AGE
nginx   k8s.io/ingress-nginx   <none>       1m21s

View the created service:

kutectl get svc -n ingress

The ing-ingress-nginx-controller should now have EXTERNAL-IP in a form of Amazon URL.This URL needs to be mapped to the domain that you have chosen. In this case I am using Amazon Route 53. A simple “A” type record is all that is needed. Other providers should work the same.

All we did in this step is create something to listen to the Ingress resource request. But there is still no Ingress resource created. Let’s do that next.

Create a yaml file with Ingress configuration, see example below:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: ingress-opensearch
  namespace: default
  annotations:
    nginx.ingress.kubernetes.io/backend-protocol: "HTTPS"
spec:
  ingressClassName: nginx
  tls:
  - hosts:
    - your-domain-here.com
    secretName: tls-for-dashboards-key-pair
  rules:
  - host: your-domain-here.com
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: dashboards-opensearch-dashboards
            port:
              number: 5601

In this case we are asking Ingress to use TLS secret name tls-for-dashboards-key-pair which doesn’t exist yet, it will be created at a later stage. Also we point to a service dashboards-opensearch-dashboards which also doesn’t exist yet.

Once the file is saved, deploy it with:

kubectl apply -f ingress.yaml

You can now view your newly created ingress resource with:

kubectl get ingress

You might notice that the ADDRESS field is empty, however this should change in the next couple of seconds/minutes and you should see the very same URL that you mapped to your hosted domain.

Summary

Up to this point, we have set our environment up. We deployed cert-manager in a fresh k8s Amazon EKS cluster. We then deployed Ingress Controller and Ingress Resource and mapped the provided URL for the load balancer to the Amazon Route 53 Domain.

In Part 2 of this blog we will use cert-manager to create a self-signed CA and then use that CA to sign several TLS certificates. As a last step, we will use Helm to deploy OpenSearch and OpenSearch Dashboards making use of these TLS certs created and managed by cert-manager.