A little while ago I wrote a blog on “Using OpenSearch on Kubernetes
”, which covered how to:
Install custom certificates
Load custom Elasticsearch configuration
Load custom Kibana configuration
Load custom security configuration
Troubleshoot common issues
This blog is going to cover the next natural step in the process of running OpenSearch in Kubernetes: using cert-manager to generate (and more importantly automatically rotate) TLS certificates. Removing the need for manual intervention when it comes to managing TLS certificates and therefore reducing the chances of human error and possible downtime. As an added bonus, since this process is done automatically, you can utilize short lived certificates, thus increasing the security of your OpenSearch deployment.
We are Going to Cover:
Deployment of cert-manager in k8s
Deployment of Ingress Controller and Resource
Creation of self-signed CA using cert-manager (Optional)
Creation of TLS certificates signed by self-signed CA using cert-manager
Using these certificates in deployment of OpenSearch/Dashboards using Helm
The technologies we are going to use are:
Amazon Elastic Kubernetes Service (EKS) Cluster
All configuration files and commands are available on GitHub: https://github.com/Anthony7774/k8s-cert-manager-opensearch
So lets begin!
The first step, of course, is to have cluster with Kubernetes running, I will be using Amazon EKS cluster, started using below command:
eksctl create cluster \
--name opensearch-cluster \
--region eu-west-1 \
--node-type t2.medium \
As I am based in Ireland,
eu-west-1 was chosen, select the region that makes sense to you.
N.B. Deleting cluster can be done with:
eksctl delete cluster --name opensearch-cluster
However, note that the PV remains on AWS and will accumulate charges if not deleted manually
1. Deployment of Cert-manager in k8s
Cert-manager is an open source tool donated by Jetstack in 2020 to the CNCF. It is a fantastic way to manage TLS certificates in Kubernetes cluster. The certificate renewal process is seamless and the configuration is very straightforward. Making this tool a must have
for any cluster using TLS certificates.
The docs are very good.
Install the cert-manager:
kubectl create namespace cert-manager
helm repo add jetstack https://charts.jetstack.io
helm repo update
helm install cert-manager jetstack/cert-manager --namespace cert-manager --version v1.7.0 --set installCRDs=true
The CRDs that are going to be installed are:
If everything works as expected you will not have to deal with these CRDs individually, however these are great for debugging cert-manager’s actions behind the scenes.
Once this is installed successfully, which should only take a minute or two, below command will display the pods now running in cert-manager namespace:
kubectl get pods --namespace cert-manager
You should see 3 pods running as follows:
NAME READY STATUS RESTARTS AGE
cert-manager-847544bbd-dnm54 1/1 Running 0 37s
cert-manager-cainjector-5c747645bf-qh98q 1/1 Running 0 37s
cert-manager-webhook-f588b48b8-tgb6r 1/1 Running 0 37s
2. Deployment of Ingress Controller and Ingress Resource
As this is a new cluster we do not have any ingress controller and therefore no
This can be verified by running the below command:
kubectl get ingressClass
You should see “No resources found”.
To install Ingress Controller, first create a yaml file requesting network-lb
nlb instead of classic
lb (which is default in Amazon cluster):
Then run below command:
helm install ing ingress-nginx/ingress-nginx \
--namespace ingress \
--version 4.0.1 \
--values ingress_controller.yaml \
Once this is installed, you can run the below command again to view the
kubectl get ingressClass
You should see something like this:
NAME CONTROLLER PARAMETERS AGE
nginx k8s.io/ingress-nginx <none> 1m21s
View the created service:
kutectl get svc -n ingress
The ing-ingress-nginx-controller should now have EXTERNAL-IP in a form of Amazon URL.This URL needs to be mapped to the domain that you have chosen. In this case I am using Amazon Route 53. A simple “A” type record is all that is needed. Other providers should work the same.
All we did in this step is create something to listen to the Ingress resource request. But there is still no Ingress resource created. Let’s do that next.
Create a yaml file with Ingress configuration, see example below:
- host: your-domain-here.com
- path: /
In this case we are asking Ingress to use TLS secret name tls-for-dashboards-key-pair which doesn’t exist yet, it will be created at a later stage. Also we point to a service dashboards-opensearch-dashboards which also doesn’t exist yet.
Once the file is saved, deploy it with:
kubectl apply -f ingress.yaml
You can now view your newly created ingress resource with:
kubectl get ingress
You might notice that the ADDRESS field is empty, however this should change in the next couple of seconds/minutes and you should see the very same URL that you mapped to your hosted domain.
Up to this point, we have set our environment up. We deployed cert-manager in a fresh k8s Amazon EKS cluster. We then deployed Ingress Controller and Ingress Resource and mapped the provided URL for the load balancer to the Amazon Route 53 Domain.
In Part 2 of this blog we will use cert-manager to create a self-signed CA and then use that CA to sign several TLS certificates. As a last step, we will use Helm to deploy OpenSearch and OpenSearch Dashboards making use of these TLS certs created and managed by cert-manager.