OpenSearch

2022-03-02

OpenSearch with cert-manager: Part 2 (Self-signed CA)

We'll dive into the details of using cert-manager to create a self-signed CA. We will sign TLS certificates with this CA and deploy them to OpenSearch.

Hello Community!

In Part 1 of this blog, we focused on setting up our environment. In particular, we deployed cert-manager in a fresh k8s Amazon EKS cluster, we deployed Ingress Controller and Ingress Resource, and we mapped the provided URL for the load balancer to the Amazon Route 53 Domain.

Here, in Part 2, we’ll dive into the details of using cert-manager to create a self-signed CA, and signing TLS certificates with this CA. Our ultimate goal is to deploy an OpenSearch and OpenSearch-Dashboards cluster, where we are going to make use of these certificates created and managed by cert-manager.

3. Creation of Self-signed CA Using cert-manager (Optional)

In this step we are finally going to start using the cert-manager.

In order to issue any certificate (CA or otherwise) we first need ClusterIssuer (Or Issuer - specific to a namespace). The clusterIssuer.yaml file for self signed ClusterIssuer is extremely short and simple:

apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: selfsigned
spec:
  selfSigned: {}

The second piece of the puzzle is the ca.yaml file with configuration of the TLS certificate that you want to be generated by this ClusterIssuer. See example below:

apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: ca-cert
  namespace: cert-manager
spec:
  isCA: true
  duration: 43800h # 5 years
  commonName: not-important.io
  secretName: ca-key-pair
  privateKey:
    algorithm: RSA
    encoding: PKCS8
    size: 4096
  issuerRef:
    name: selfsigned
    kind: ClusterIssuer
    group: cert-manager.io

Note the following:

commonName is not important as it will be self-signed

secretName is the location where the secret “will be created” after applying this file.

issuerRef.name is where we link this certificate to the ClusterIssuer in the first file.

privateKey.encoding: PKCS8 is the only support format, the default in cert-manager is PKCS1, which would need to be converted after creation. (At the time of writing this blog there is a ticket created to support PKCS1 (link https://github.com/opensearch-project/security/issues/1573))

Apply these files with :

kubectl apply -f clusterIssuer.yaml
kubectl apply -f ca.yaml

You can now examine the secrets created:

kubectl get secret -n cert-manager

You should see ca-key-pair as one of the secrets of type TLS.

You can also view the certificate itself using below command:

kubectl get secret ca-key-pair -o yaml -n cert-manager

Note that although the content of the tls.crt and ca.crt is base64 encoded, you should be able to see even without decoding that they are identical, this is because this certificate is self signed.

4. Creation of TLS Certificates Signed by Self-signed CA Using cert-manager

Now we are on to the most interesting part. In this section we create actual certificates that will be used by OpenSearch/Dashboards.

Similarly to the previous step, we are going to create ClusterIssuer.yaml (pointing to the already existing CA) and certificates.yaml (pointing to this ClusterIssuer).

Again the config for ClusterIssuer is rather short:

apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: cluster-issuer-from-ca
spec:
  ca:
    secretName: ca-key-pair # this is the previously created secret

N.B. As you might have noticed, this secret did not need to be generated by cert-manager specifically, this could be any CA that you already have loaded in as a secret. Therefore that step was marked as optional.

opensearch_certificate.yaml:

apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: tls-for-opensearch
  namespace: default
spec:
  isCA: false
  duration: 2160h # 90d
  renewBefore: 360h # 15d
  commonName: your-domain-here.com
  dnsNames:
  - your-domain-here.com
  - www.your-domain-here.com
  secretName: tls-for-opensearch-key-pair
  privateKey:
    algorithm: RSA
    encoding: PKCS8
    size: 2048
  usages:
    - server auth
    - client auth
  issuerRef:
    name: cluster-issuer-from-ca
    kind: ClusterIssuer
    group: cert-manager.io

dashboards_certificate.yaml:

apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: tls-for-dashboards
  namespace: default
spec:
  secretName: tls-for-dashboards-key-pair
  issuerRef:
    name: cluster-issuer-from-ca
    kind: ClusterIssuer
    group: cert-manager.io
  commonName: your-domain-here.com
  dnsNames:
  - your-domain-here.com
  - www.your-domain-here.com
  privateKey:
    algorithm: RSA
    encoding: PKCS8
    size: 2048
  usages:
    - client auth
    - digital signature
    - key encipherment

Most of this config should probably already be familiar. This will create 2 new TLS type secrets:

tls-for-opensearch-key-pair

tls-for-dashboards-key-pair

There are also three additional secrets needed by OpenSearch Dashboards, see below:

apiVersion: v1
kind: Secret
metadata:
  name: opensearch-dashboards-account
data:
  username: a2liYW5hc2VydmVyCg== # base64 encoded values
  password: a2liYW5hc2VydmVyCg==
  cookie: b3BlbnNlYXJjaC1kYXNoYm9hcmRzLWNvb2tpZXBhc3N3b3JkCg==

Apply all 4 files with "kubectl apply -f ."

You should now have 3 new secrets:

kubectl get secret

NAME                            TYPE                                  DATA   AGE
default-token-hgx8v             kubernetes.io/service-account-token   3      85m
opensearch-dashboards-account   Opaque                                3      2s
tls-for-dashboards-key-pair     kubernetes.io/tls                     3      1s
tls-for-opensearch-key-pair     kubernetes.io/tls                     3      1s

You can examine the certificates the same way we did previously

kubectl get secret tls-for-dashboards-key-pair -o yaml

These certificates are now fully managed by cert-manager and will be renewed at the set period (renewBefore).

5. Using these Certificates in Deployment of OpenSearch/Dashboards Using Helm

As the detailed description on how to run OpenSearch using Helm was already provided in my previous blog we will not go into how to configure everything via values.yaml. (However example is available in my GitHub as previously mentioned). The configuration is exactly the same, as the only differences were already configured in the above four steps. Therefore the only part left to do, is to actually deploy OpenSearch and OpenSearch Dashboards

helm install --values=values.yaml opensearch opensearch-<tag>.tgz
helm install --values=values.yaml dashboards opensearch-dashboards-<tag>.tgz

Prior to attempting to access via the domain, I would recommend checking the OpenSearch Dashboards logs to see if the service is running.

kubectl logs dashboards-opensearch-dashboards

The line that we are looking for is:

"http server running at https://0.0.0.0:5601"

Once this is visible in the logs, you can connect with the domain you are using and you should be able to access OpenSearch Dashboards

Certificate Hot Reload API:

When the certificates are renewed, they will be automatically loaded in the pods, however the security plug-in does not automatically pick up the changes.

Therefore a “Cert Hot Reload” API needs to be called with admin certificate. At the time of writing this, the API is still using the old version (_opendistro/_security)

If admin certificate is loaded into the pods in directory /usr/share/opensearch/config/certs/admin/ the command to view loaded certificates would be:

curl -k -XGET "https://127.0.0.1:9200/_opendistro/_security/api/ssl/certs" --cacert "/usr/share/opensearch/config/certs/admin/ca.crt" --cert "/usr/share/opensearch/config/certs/admin/tls.crt" --key "/usr/share/opensearch/config/certs/admin/tls.key"

Similarly http certificates reload API:

curl -k -XPUT "https://127.0.0.1:9200/_opendistro/_security/api/ssl/http/reloadcerts" --cacert "/usr/share/opensearch/config/certs/admin/ca.crt" --cert "/usr/share/opensearch/config/certs/admin/tls.crt" --key "/usr/share/opensearch/config/certs/admin/tls.key"

And transport certificates reload API:

curl -k -XPUT "https://127.0.0.1:9200/_opendistro/_security/api/ssl/transport/reloadcerts" --cacert "/usr/share/opensearch/config/certs/admin/ca.crt" --cert "/usr/share/opensearch/config/certs/admin/tls.crt" --key "/usr/share/opensearch/config/certs/admin/tls.key"

Conclusion

In this blog we deployed cert-manager in fresh k8s Amazon EKS cluster. We then deployed Ingress Controller and Ingress Resource and mapped the provided URL for the load balancer to the Amazon Route 53 Domain. After this, using cert-manager we created self-signed CA and used that CA to sign several TLS certificates, which will now be renewed by cert-manager. As a last step we deployed OpenSearch and OpenSearch Dashboards using Helm and tested the setup.

In a future blog I am hoping to demonstrate how to utilize commercial certificates by linking cert-manager to Let’s Encrypt and therefore avoiding the certificate not trusted warning that we currently get with self-signed CA.