In Part 1
of this blog, we focused on setting up our environment. In particular, we deployed cert-manager in a fresh k8s Amazon EKS cluster, we deployed Ingress Controller and Ingress Resource, and we mapped the provided URL for the load balancer to the Amazon Route 53 Domain.
Here, in Part 2, we’ll dive into the details of using cert-manager to create a self-signed CA, and signing TLS certificates with this CA. Our ultimate goal is to deploy an OpenSearch and OpenSearch-Dashboards cluster, where we are going to make use of these certificates created and managed by cert-manager.
3. Creation of Self-signed CA Using cert-manager (Optional)
In this step we are finally going to start using the cert-manager.
In order to issue any certificate (CA or otherwise) we first need
Issuer - specific to a namespace). The clusterIssuer.yaml file for self signed
ClusterIssuer is extremely short and simple:
The second piece of the puzzle is the ca.yaml file with configuration of the TLS certificate that you want to be generated by this
ClusterIssuer. See example below:
duration: 43800h # 5 years
Note the following:
commonName is not important as it will be self-signed
secretName is the location where the secret “will be created” after applying this file.
issuerRef.name is where we link this certificate to the
ClusterIssuer in the first file.
privateKey.encoding: PKCS8 is the only support format, the default in cert-manager is PKCS1, which would need to be converted after creation. (At the time of writing this blog there is a ticket created to support PKCS1 (link https://github.com/opensearch-project/security/issues/1573))
Apply these files with :
kubectl apply -f clusterIssuer.yaml
kubectl apply -f ca.yaml
You can now examine the secrets created:
kubectl get secret -n cert-manager
You should see ca-key-pair as one of the secrets of type TLS.
You can also view the certificate itself using below command:
kubectl get secret ca-key-pair -o yaml -n cert-manager
Note that although the content of the tls.crt and ca.crt is base64 encoded, you should be able to see even without decoding that they are identical, this is because this certificate is self signed.
4. Creation of TLS Certificates Signed by Self-signed CA Using cert-manager
Now we are on to the most interesting part. In this section we create actual certificates that will be used by OpenSearch/Dashboards.
Similarly to the previous step, we are going to create ClusterIssuer.yaml (pointing to the already existing CA) and certificates.yaml (pointing to this
Again the config for
ClusterIssuer is rather short:
secretName: ca-key-pair # this is the previously created secret
N.B. As you might have noticed, this secret did not need to be generated by cert-manager specifically, this could be any CA that you already have loaded in as a secret. Therefore that step was marked as optional.
duration: 2160h # 90d
renewBefore: 360h # 15d
- server auth
- client auth
- client auth
- digital signature
- key encipherment
Most of this config should probably already be familiar.
This will create 2 new TLS type secrets:
There are also three additional secrets needed by OpenSearch Dashboards, see below:
username: a2liYW5hc2VydmVyCg== # base64 encoded values
Apply all 4 files with
"kubectl apply -f ."
You should now have 3 new secrets:
kubectl get secret
NAME TYPE DATA AGE
default-token-hgx8v kubernetes.io/service-account-token 3 85m
opensearch-dashboards-account Opaque 3 2s
tls-for-dashboards-key-pair kubernetes.io/tls 3 1s
tls-for-opensearch-key-pair kubernetes.io/tls 3 1s
You can examine the certificates the same way we did previously
kubectl get secret tls-for-dashboards-key-pair -o yaml
These certificates are now fully managed by cert-manager and will be renewed at the set period (renewBefore).
5. Using these Certificates in Deployment of OpenSearch/Dashboards Using Helm
As the detailed description on how to run OpenSearch using Helm was already provided in my previous blog
we will not go into how to configure everything via values.yaml. (However example is available in my GitHub as previously mentioned). The configuration is exactly the same, as the only differences were already configured in the above four steps. Therefore the only part left to do, is to actually deploy OpenSearch and OpenSearch Dashboards
helm install --values=values.yaml opensearch opensearch-<tag>.tgz
helm install --values=values.yaml dashboards opensearch-dashboards-<tag>.tgz
Prior to attempting to access via the domain, I would recommend checking the OpenSearch Dashboards logs to see if the service is running.
kubectl logs dashboards-opensearch-dashboards
The line that we are looking for is:
"http server running at https://0.0.0.0:5601"
Once this is visible in the logs, you can connect with the domain you are using and you should be able to access OpenSearch Dashboards
Certificate Hot Reload API:
When the certificates are renewed, they will be automatically loaded in the pods, however the security plug-in does not automatically pick up the changes.
Therefore a “Cert Hot Reload” API needs to be called with admin certificate.
At the time of writing this, the API is still using the old version (_opendistro/_security)
If admin certificate is loaded into the pods in directory /usr/share/opensearch/config/certs/admin/ the command to view loaded certificates would be:
curl -k -XGET "https://127.0.0.1:9200/_opendistro/_security/api/ssl/certs" --cacert "/usr/share/opensearch/config/certs/admin/ca.crt" --cert "/usr/share/opensearch/config/certs/admin/tls.crt" --key "/usr/share/opensearch/config/certs/admin/tls.key"
Similarly http certificates reload API:
curl -k -XPUT "https://127.0.0.1:9200/_opendistro/_security/api/ssl/http/reloadcerts" --cacert "/usr/share/opensearch/config/certs/admin/ca.crt" --cert "/usr/share/opensearch/config/certs/admin/tls.crt" --key "/usr/share/opensearch/config/certs/admin/tls.key"
And transport certificates reload API:
curl -k -XPUT "https://127.0.0.1:9200/_opendistro/_security/api/ssl/transport/reloadcerts" --cacert "/usr/share/opensearch/config/certs/admin/ca.crt" --cert "/usr/share/opensearch/config/certs/admin/tls.crt" --key "/usr/share/opensearch/config/certs/admin/tls.key"
In this blog we deployed cert-manager in fresh k8s Amazon EKS cluster. We then deployed Ingress Controller and Ingress Resource and mapped the provided URL for the load balancer to the Amazon Route 53 Domain.
After this, using cert-manager we created self-signed CA and used that CA to sign several TLS certificates, which will now be renewed by cert-manager.
As a last step we deployed OpenSearch and OpenSearch Dashboards using Helm and tested the setup.
In a future blog I am hoping to demonstrate how to utilize commercial certificates by linking cert-manager to Let’s Encrypt and therefore avoiding the certificate not trusted warning that we currently get with self-signed CA.