How to configure PKI authentication in Elasticsearch

Introduction

In many organizations it is forbidden to store usernames and passwords in source code or configuration files, which means that the basic native realm username and password authentication in Elasticsearch cannot be used for server-to-server authentication. In such cases one alternative is to use PKI Authentication for authenticating to an Elasticsearch cluster.

Because the documentation for setting up PKI authentication can be overwhelming, in this blog post we provide a step-by-step guide on how to use client TLS/SSL certificates to authenticate to an Elasticsearch cluster. We then show how to configure Kibana to authenticate to an Elasticsearch cluster using only a client certificate, which removes the requirement for embedding a username and password in the “kibana.yml” configuration file.

Enabling security

In order to enable TLS/SSL it is necessary to have either a gold or platinum subscription, or to enable a trial license via kibana or enable a trial license via the API.

Once you have enabled a license, you need to enable security. This is done in the elasticsearch.yml file with the following line:

xpack.security.enabled: true

Note that Elasticsearch has two levels of communications, the transport layer and http layer. The transport layer is used for internal communications between Elasticsearch nodes, and the http layer is used for communications from clients to the Elasticsearch cluster.

Transport layer TLS/SSL encryption

The transport layer is used for communication between nodes within the Elasticsearch cluster. Because each node in an Elasticsearch cluster is both a client and a server to other nodes in the cluster, all transport certificates must be created as both client and server certificates. If TLS/SSL certificates do not have “Extended Key Usage” defined, then they are already both client and server certificates. If transport certificates do have an “Extended Key Usage” section, which is usually the case for “real” certificates used in corporate environments, then they must explicitly enable use as both client and server certificates on the transport layer.

Note that Elasticsearch comes with a utilitity called elasticsearch-certutil that can be used for generating self-signed certificates that can be used for encrypting internal communications within an Elasticsearch cluster.

Using elasticsearch-certutil for encrypting transport communications is a valid option, as the corporate CA will not be able to create certificates that would allow a new node to join the Elasticsearch cluster — only someone with access to the newly created self-signed root CA’s private key will be able create certificates that would allow new nodes to join the cluster.

The following commands can be used for generating certificates that can be used for transport communications, as described in this page on Encrypting Communications in Elasticsearch:

bin/elasticsearch-certutil ca
ENTER ENTER
bin/elasticsearch-certutil cert --ca elastic-stack-ca.p12
ENTER ENTER ENTER

Once the above commands have been executed, you will have certificates that can be used for encrypting communications, which will be specified in the elasticsearch.yml file as follows:

xpack.security.transport.ssl.enabled: true
xpack.security.transport.ssl.verification_mode: certificate
xpack.security.transport.ssl.keystore.path: path/elastic-certificates.p12
xpack.security.transport.ssl.truststore.path: path/elastic-certificates.p12

Http layer TLS/SSL encryption

For Http communications, the Elasticsearch nodes will only act as servers and therefore can use Server certificates – i.e. Http TLS/SSL certificates do not need to enable Client authentication.

In many cases, certificates for Http communications would be signed  by a corporate CA. It is worth noting that the certificates used for encrypting Http communications are totally independent from the certificates that are used for transport communications.

To reduce the number of steps in this blog, we use the same certificates for Http communications as we use for the transport communications. These are specified in the elasticsearch.yml file as follows:

xpack.security.http.ssl.enabled: true
xpack.security.http.ssl.keystore.path: path/elastic-certificates.p12
xpack.security.http.ssl.truststore.path: path/elastic-certificates.p12
xpack.security.http.ssl.client_authentication: optional

Enabling PKI authentication

As discussed in Configuring a PKI Realm, the following must be added to the elasticsearch.yml file to allow PKI authentication.

xpack.security.authc.realms.pki1.type: pki

Combined changes to elasticsearch.yml

Once the above steps have been followed, you should have a section in your elasticsearch.yml file that looks similar to the following:

xpack.security.enabled: true

xpack.security.transport.ssl.enabled: true
xpack.security.transport.ssl.verification_mode: certificate
xpack.security.transport.ssl.keystore.path: path/elastic-certificates.p12
xpack.security.transport.ssl.truststore.path: path/elastic-certificates.p12

xpack.security.http.ssl.enabled: true
xpack.security.http.ssl.keystore.path: path/elastic-certificates.p12
xpack.security.http.ssl.truststore.path: path/elastic-certificates.p12
xpack.security.http.ssl.client_authentication: optional

xpack.security.authc.realms.pki1.type: pki

Creating a client certificate

Certificates that will be used for PKI authentication must be signed by the same CA as the certificates that are used for encrypting the Http layer communications. Normally these are signed by an official CA within an organization.

However, as I do not have an official CA, and because we used a self signed CA to sign certificates used in the Http layer ssl configuration, I am also signing client certificates with our self-signed CA. We can create such a certificate for client authentication as follows:

bin/elasticsearch-certutil cert --ca elastic-stack-ca.p12 \
-name "CN=something,OU=Consulting Team,DC=mydomain,DC=com"
ENTER
client.p12 ENTER
ENTER

The above will create a file called client.p12, which contains all of the information required for PKI authentication to your Elasticsearch cluster. However, in order to use this certificate we need to break it into its private key, public certificate, and CA certificate. This can be done with the following commands:

openssl pkcs12 -in client.p12 -nocerts -nodes | \
sed -ne '/-BEGIN PRIVATE KEY-/,/-END PRIVATE KEY-/p' > client.key
ENTER
openssl pkcs12 -in client.p12 -clcerts -nokeys | \
sed -ne '/-BEGIN CERTIFICATE-/,/-END CERTIFICATE-/p' > client.cer
ENTER

openssl pkcs12 -in client.p12 -cacerts -nokeys -chain | \
sed -ne '/-BEGIN CERTIFICATE-/,/-END CERTIFICATE-/p' > client-ca.cer
ENTER

Which should produce three files,

  1. “client.key” – The private key.
  2. “client.cer” – The public certificate.
  3. “client-ca.cer” – The certificate of the CA that signed the public certificate.

PKI Authentication

We can now use these three files to test PKI authentication to the cluster using curl as demonstrated with the call to the authenticate API below.

curl https://localhost:9200/_xpack/security/_authenticate?pretty \
--key client.key --cert client.cer --cacert client-ca.cer -k -v

Be sure to replace “localhost” with the name of a node in your Elasticsearch cluster. Also note that the -k option is required as we did not create certificates with the hostnames specified, and therefore hostname verification must be turned off.

The above command should respond with something similar to the following:

{
  "username" : "something",
  "roles" : [ ],
  "full_name" : null,
  "email" : null,
  "metadata" : {
    "pki_dn" : "CN=something, OU=Consulting Team, DC=mydomain, DC=com"
  },
  "enabled" : true
}

Notice that the “roles” is currently empty which means that although we have authenticated to Elasticearch, we are not authorized to perform any actions. This authentication is allowed, because the client certificate that we sent to the cluster was signed by the same CA as the Http TLS/SSL certificates used by the Elasticsearch nodes. Now that we are authenticated, we need to authorize this user to be able to do something.

The “pki_dn” value returned from the authenticate API can now be used to configure the roles that will be assigned to this certificate. This can be done by copying the “pki_dn” value into the “dn” field as follows. This command should be executed by a user with sufficient privileges to modify roles, such as the “elastic” user:

PUT _xpack/security/role_mapping/my_dummy_user
{
  "roles" : [ "kibana_user", "watcher_user" ],
  "rules" : { "field" : { "dn" : "CN=something, OU=Consulting Team, DC=mydomain, DC=com" } },
  "enabled": true
}

Now that we have assigned “kibana_user” and “watcher_user” roles to this certificate, we can again execute a call to the authenticate API as follows:

curl https://localhost:9200/_xpack/security/_authenticate?pretty \
--key client.key --cert client.cer --cacert client-ca.cer -k -v

And we should see the following response, which indicates that we now have the roles that we have just assigned to this certificate.

{
  "username" : "something",
  "roles" : [
    "kibana_user",
    "watcher_user"
  ],
  "full_name" : null,
  "email" : null,
  "metadata" : {
    "pki_dn" : "CN=something, OU=Consulting Team, DC=mydomain, DC=com"
  },
  "enabled" : true
}

Using PKI to secure communications from Kibana to the Elasticsearch cluster

Once you have tested your client-side certificates, you can assign the “kibana_system” role to a certificate as follows. Note that this will overwrite any previous roles assigned to this certificate:

PUT _xpack/security/role_mapping/kibana_certificate
{
  "roles" : [ "kibana_system" ],
  "rules" : { "field" : { "dn" : "CN=something, OU=Consulting Team, DC=mydomain, DC=com" } },
  "enabled": true
}

We can now remove the following lines from our “kibana.yml” file:

elasticsearch.username: "kibana"
elasticsearch.password: "XXXXXX"

And add the following line for authenticating with the client certificate:

xpack.security.enabled: true
elasticsearch.ssl.certificate: path/client.cer
elasticsearch.ssl.key: path/client.key
elasticsearch.ssl.certificateAuthorities: path/client-ca.cer
elasticsearch.ssl.verificationMode: certificate

You can now restart kibana, and it should authenticate to your Elasticsearch cluster, without the need for an embedded username and password!

Conclusion

In this blog post, we have demonstrated how to enable PKI authentication and how PKI authentication can be used instead of usernames and passwords to authenticate to an Elasticsearch cluster. We then demonstrated how using PKI authentication can eliminate the need for users and passwords in the “kibana.yml” file.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s