OpenSearch

2022-03-09

OpenSearch with cert-manager: Part 3 (Let's Encrypt)

In this 3rd part, we are going to cover using commercial certificates from Let's Encrypt to bootstrap the OpenSearch/Dashboards cluster and therefore removing the need to generate self-signed CA.

Hello Community!

In Part 1 of this series, we focused on setting up our environment. In particular, we deployed cert-manager in a fresh k8s Amazon EKS cluster, deployed Ingress Controller and Ingress Resource and mapped the provided URL for the load balancer to the Amazon Route53 Domain.

In Part 2 we covered creating and using self-signed CA with the help of cert-manager. Generating certificates signed using that CA and deploying OpenSearch/Dashboards using Helm charts.

In this 3rd part, we are going to cover using commercial certificates from Let’s Encrypt to bootstrap the OpenSearch/Dashboards cluster and therefore removing the need to generate self-signed CA and more importantly no longer needing to manually bypass the security warnings we see in the browser due to untrusted certificates.

As previously mentioned configuration files are available at Github.

If you were following the previous two parts, currently you should have OpenSearch/Dashboards running inside kubernetes using self-signed CA. In order to follow along please run below commands to clean up the environment:

helm delete dashboards
helm delete opensearch
kubectl delete -f example-2
kubectl delete -f example-1

Make sure that there are no leftover certificates/secrets from previous exercise (self-signed CA and certificates). Your state should match the following:

kubectl get secret
NAME                  TYPE                                  DATA   AGE
default-token-28mww   kubernetes.io/service-account-token   3      9m

6. Generating Commercial Let’s Encrypt Certificates Using cert-manager

The inner workings of the Let’s Encrypt are outside the scope of this blog, however the docs are quite good.

In order to obtain certificates you need to prove that you indeed own the domain. Luckily for us this process is seamless with cert-manager. There are only two ways that you can prove the ownership:

Provisioning a DNS record

Provisioning a HTTP resource under a well-known URI.

In our case we are going to use DNS method (further details as to why you might want to use one over the other are described here).

Similarly to the clusterIssuer file we have seen in Part 2 of this blog, we will create clusterIssuer using DNS option. As we are using AWS, we can avail of the Route53 option seen in file lets_encrypt/clusterIssuer_dns.yaml:

apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: letsencrypt-prod
spec:
  acme:
    #server: https://acme-staging-v02.api.letsencrypt.org/directory (STAGING)
    server: https://acme-v02.api.letsencrypt.org/directory # (PROD)
    email: [email protected]
    privateKeySecretRef:
      name: letsencrypt-prod
    solvers:
      - selector:
          dnsZones:
            - "eliatra-support.co.uk"
        dns01:
          route53:
            region: eu-west-1
            accessKeyID: AKI...
            secretAccessKeySecretRef:
              name: my-aws-secret
              key: token

As you can see this option also requires the AWS accessKey, let’s create this using file lets_encrypt/aws_secret.yaml:

apiVersion: v1
kind: Secret
metadata:
  name: my-aws-secret
  namespace: cert-manager
type: Opaque
data:
  token: ....

Next we need to create secrets needed by OpenSearch Dashboards (username, password, cookies):

lets_encrypt/secret_dashboards.yaml:

apiVersion: v1
kind: Secret
metadata:
  name: opensearch-dashboards-account
data:
  username: a2liYW5hc2VydmVyCg==
  password: a2liYW5hc2VydmVyCg==
  cookie: b3BlbnNlYXJjaC1kYXNoYm9hcmRzLWNvb2tpZXBhc3N3b3JkCg==

The last two files are very similar to previously discussed certificate files, we simply request the necessary certificates using the above mentioned cluster issuers:

lets_encrypt/certificate_kibana.yaml

apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: tls-for-dashboards
  namespace: default
spec:
  secretName: tls-for-dashboards
  issuerRef:
    name: letsencrypt-prod
    kind: ClusterIssuer
    group: cert-manager.io
  commonName: eliatra-support.co.uk
  dnsNames:
  - eliatra-support.co.uk
  privateKey:
    algorithm: RSA
    encoding: PKCS8
    size: 2048
  usages:
    - server auth
    - client auth

lets_encrypt/certificate.yaml

apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: tls-for-opensearch
  namespace: default
spec:
  secretName: tls-for-opensearch
  issuerRef:
    name: letsencrypt-prod
    kind: ClusterIssuer
    group: cert-manager.io
  commonName: eliatra-support.co.uk
  dnsNames:
  - eliatra-support.co.uk
  privateKey:
    algorithm: RSA
    encoding: PKCS8
    size: 2048
  usages:
    - server auth
    - client auth

Apply all of these using below command:

kubectl apply -f lets_encrypt/.

If you run kubectl get cert you might notice that the READY status is ‘False’, if everything was configured correctly, this should change to ‘True’ within minutes and you should see secrets created as below:

kubectl get secrets

NAME                  TYPE                                  DATA   AGE
default-token-28mww   kubernetes.io/service-account-token   3      9m
tls-for-dashboards    kubernetes.io/tls                     2      5m
tls-for-opensearch    kubernetes.io/tls                     2      6m

You might also notice on the AWS Management Console (Route53 -> Hosted Zones dashboard), a new ‘TXT’ entry with a record name _acme-challenge., this is the record used to prove the ownership of your domain.

7. Deploying OpenSearch/Dashboards with Let’s Encrypt Certificates

The last step is to deploy OpenSearch and OpenSearch Dashboards using helm.

The OpenSearch configuration is almost identical to configuration used with self-signed CA, however in this case pemcert_filepath and pemtrustedcas_filepath will point to the same tls.crt file, as there are actually three certificates provided within this file by Let’s Encrypt.

Therefore the configuration extract from values.yaml file is:

plugins:
      security:
        nodes_dn:
           - 'CN=eliatra-support.co.uk'
        ssl:
          transport:
            pemcert_filepath: certs/tls.crt
            pemkey_filepath: certs/tls.key
            pemtrustedcas_filepath: certs/tls.crt
            enforce_hostname_verification: false
          http:
            enabled: true
            pemcert_filepath: certs/tls.crt
            pemkey_filepath: certs/tls.key
            pemtrustedcas_filepath: certs/tls.crt
        allow_unsafe_democertificates: false

(The full configuration file is located at lets_encrypt/charts/opensearch/values.yaml)

Run the below commands to package and deploy OpenSearch:

cd lets_encrypt/charts/opensearch/
helm package .
helm install --values values.yaml opensearch opensearch-1.7.1.tgz

Deploying OpenSearch Dashboards is very similar, however, there is one additional complexity involved if you are using opensearch.ssl.verificationMode: certificate opensearch.ssl.certificateAuthorities field needs to be pointing to top level self-signed CA which is not included as one of the 3 certificates provided by Let’s Encrypt. Therefore there is some additional work needed to get this working.

We need to download the required certificate and map it in the container. Although this may seem like a challenging task, this is indeed a rather straightforward matter thanks to initContainers:

extraInitContainers: 
  - name: init-script-downloader
    image: appropriate/curl
    args:
      - "-k"
      - "-o"
      - "/tmp/ca-root.pem"
      - "https://letsencrypt.org/certs/isrgrootx1.pem" 
    volumeMounts:
    - name: ca-cert # mount the volume where downloaded file will be saved
      mountPath: /tmp

With the above configuration, prior to OpenSearch Dashboards container starting, initContainer will download the needed certificate and save it in a volume which will then be mapped to the main OpenSearch Dashboards container. (Again full file is available on github at lets_encrypt/charts/opensearch-dashboards/values.yaml)

Run the below commands to package and deploy OpenSearch Dashboards:

cd lets_encrypt/charts/opensearch-dashboards/
helm package .
helm install --values values.yaml opensearch opensearch-dashboards-1.2.0.tgz

Ensure that all pods are running:

kubectl get pods
NAME                                               READY   STATUS    RESTARTS   AGE
dashboards-opensearch-dashboards-6bf8d46c4-tx2xd   1/1     Running   0          41s
opensearch-cluster-master-0                        1/1     Running   0          2m9s
opensearch-cluster-master-1                        1/1     Running   0          2m9s
opensearch-cluster-master-2                        1/1     Running   0          2m9s

Prior to accessing the url in a browser, I would recommend to examine the logs on the Opensearch Dashboards pods to check if service is running:

kubectl logs dashboards-opensearch-dashboards-6bf8d46c4-tx2xd --follow

The message you are looking for is "Server running at https://0.0.0.0:5601"

Now access your domain via browser and if everything was configured correctly, you should see the OpenSearch Dashboards login screen with a little lock icon next to the url bar.

Conclusion

Having bootstrapped the cluster with Let’s Encrypt certificates using cert-manager, we can now have peace of mind due to automatic rotation of the certificates, reducing the need for manual intervention and possible downtime and have browsers automatically trust these certificates.

There is still, of course, a requirement to call the certificate hot reload API as described in Part 2 of the series, to reload the new certificates after the automatic rotation.

Summary

In this series we started with a vanilla instance of EKS cluster. We began by deploying necessary ingress resources and cert-manager, which in turn installed the necessary CRDs for certificate generation.

In the second part using cert-manager we created a self-signed CA and generated required certificates signed by that CA. We then deployed OpenSearch/Dashboards using those certificates and were able to access OpenSearch Dashboards via browser, however were getting the untrusted warning in the browser due to a self-signed certificate.

In the last part of the series, we went a step further and linked up cert-manager with Let’s Encrypt. We used the DNS method to prove ownership of the domain and generated the necessary certificates. We then deployed a new version of OpenSearch/Dashboards using these certificates and were able to access the OpenSearch Dashboards via browser without any certificate warning as the signing CA from Let’s Encrypt is already trusted by the browsers.