OpenSearch with cert-manager: Part 1 (Setup)

OpenSearch

2022-02-23

OpenSearch with cert-manager: Part 1 (Setup)

Using cert-manager in Kubernetes to generate (and more importantly automatically rotate) TLS certificates on OpenSearch and Dashboards..

A little while ago I wrote a blog on “Using OpenSearch on Kubernetes”, which covered how to:
    Install custom certificates
    Load custom Elasticsearch configuration
    Load custom Kibana configuration
    Load custom security configuration
    Troubleshoot common issues
This blog is going to cover the next natural step in the process of running OpenSearch in Kubernetes: using cert-manager to generate (and more importantly automatically rotate) TLS certificates. Removing the need for manual intervention when it comes to managing TLS certificates and therefore reducing the chances of human error and possible downtime. As an added bonus, since this process is done automatically, you can utilize short lived certificates, thus increasing the security of your OpenSearch deployment.

We are Going to Cover:

Part 1
    Deployment of cert-manager in k8s
    Deployment of Ingress Controller and Resource
Part 2
    Creation of self-signed CA using cert-manager (Optional)
    Creation of TLS certificates signed by self-signed CA using cert-manager
    Using these certificates in deployment of OpenSearch/Dashboards using Helm
The technologies we are going to use are:
    Amazon Elastic Kubernetes Service (EKS) Cluster
    Amazon Route53
    cert-manager
    Ingress NGINX
    Helm
All configuration files and commands are available on GitHub: https://github.com/Anthony7774/k8s-cert-manager-opensearch
So lets begin!
The first step, of course, is to have cluster with Kubernetes running, I will be using Amazon EKS cluster, started using below command:
eksctl create cluster \
--name opensearch-cluster \
--region eu-west-1 \
--node-type t2.medium \
--nodes 3
As I am based in Ireland, eu-west-1 was chosen, select the region that makes sense to you.
N.B. Deleting cluster can be done with:
eksctl delete cluster --name opensearch-cluster
However, note that the PV remains on AWS and will accumulate charges if not deleted manually

1. Deployment of Cert-manager in k8s

Cert-manager is an open source tool donated by Jetstack in 2020 to the CNCF. It is a fantastic way to manage TLS certificates in Kubernetes cluster. The certificate renewal process is seamless and the configuration is very straightforward. Making this tool a must have for any cluster using TLS certificates. The docs are very good.
Install the cert-manager:
kubectl create namespace cert-manager
helm repo add jetstack https://charts.jetstack.io
helm repo update
helm install cert-manager jetstack/cert-manager --namespace cert-manager --version v1.7.0 --set installCRDs=true
The CRDs that are going to be installed are:
    Issuer
    ClusterIssuer
    Certificate
    CertificateRequest
    Order
    Challenge
If everything works as expected you will not have to deal with these CRDs individually, however these are great for debugging cert-manager’s actions behind the scenes.
Once this is installed successfully, which should only take a minute or two, below command will display the pods now running in cert-manager namespace
kubectl get pods --namespace cert-manager
You should see 3 pods running as follows:
NAME                                       READY   STATUS    RESTARTS   AGE
cert-manager-847544bbd-dnm54               1/1     Running   0          37s
cert-manager-cainjector-5c747645bf-qh98q   1/1     Running   0          37s
cert-manager-webhook-f588b48b8-tgb6r       1/1     Running   0          37s

2. Deployment of Ingress Controller and Ingress Resource

As this is a new cluster we do not have any ingress controller and therefore no ingressClass available.
This can be verified by running the below command
kubectl get ingressClass
You should see “No resources found”
To install Ingress Controller, first create a yaml file requesting network-lb nlb instead of classic lb (which is default in Amazon cluster):
---
controller:
  service:
    annotations:
      service.beta.kubernetes.io/aws-load-balancer-type: nlb
Then run below command:
helm install ing ingress-nginx/ingress-nginx \
  --namespace ingress \
  --version 4.0.1 \
  --values ingress_controller.yaml \
  --create-namespace
Once this is installed, you can run the below command again to view the ingressClass created:
kubectl get ingressClass
You should see something like this:
NAME    CONTROLLER             PARAMETERS   AGE
nginx   k8s.io/ingress-nginx   <none>       1m21s
View the created service:
kutectl get svc -n ingress
The ing-ingress-nginx-controller should now have EXTERNAL-IP in a form of Amazon URL.This URL needs to be mapped to the domain that you have chosen. In this case I am using Amazon Route 53. A simple “A” type record is all that is needed. Other providers should work the same.
All we did in this step is create something to listen to the Ingress resource request. But there is still no Ingress resource created. Let’s do that next.
Create a yaml file with Ingress configuration, see example below:
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: ingress-opensearch
  namespace: default
  annotations:
    nginx.ingress.kubernetes.io/backend-protocol: "HTTPS"
spec:
  ingressClassName: nginx
  tls:
  - hosts:
    - your-domain-here.com
    secretName: tls-for-dashboards-key-pair
  rules:
  - host: your-domain-here.com
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: dashboards-opensearch-dashboards
            port:
              number: 5601
In this case we are asking Ingress to use TLS secret name tls-for-dashboards-key-pair which doesn’t exist yet, it will be created at a later stage. Also we point to a service dashboards-opensearch-dashboards which also doesn’t exist yet.
Once the file is saved, deploy it with:
kubectl apply -f ingress.yaml
You can now view your newly created ingress resource with:
kubectl get ingress
You might notice that the ADDRESS field is empty, however this should change in the next couple of seconds/minutes and you should see the very same URL that you mapped to your hosted domain.

Summary

Up to this point, we have set our environment up. We deployed cert-manager in a fresh k8s Amazon EKS cluster. We then deployed Ingress Controller and Ingress Resource and mapped the provided URL for the load balancer to the Amazon Route 53 Domain.
In Part 2 of this blog we will use cert-manager to create a self-signed CA and then use that CA to sign several TLS certificates. As a last step, we will use Helm to deploy OpenSearch and OpenSearch Dashboards making use of these TLS certs created and managed by cert-manager.
Get in touch with us
Send us a message and talk to the team today.
Need help with your setup?
OpenSearch or Elastic Stack – We’ve got you covered.