Improve The CrowdSec Multi-Server Installation With HTTPS Between Agents
This article is a follow-up from the Crowdsec multi-server setup. It applies to a configuration with at least two servers (referred to as
server-1 and one of
To address security issues posed by clear http communication in our previous crowdsec multi-server installation, we propose solutions to achieve communication between Crowdsec agents over encrypted channels. On top of that, the third solution allows
server-3 to trust
server-1 identity, and avoid man-in -the -middle attacks.
Using self-signed certificates
Create the certificate
First we have to create a certificate. This can be achieved with the following one-liner.
openssl req -x509 -newkey rsa:4096 -keyout encrypted-key.pem -out cert.pem -days 365 -addext "subjectAltName = IP:172.31.100.242"
For now crowdsec is not able to ask for the passphrase of the private key when starting. Thus we have the choice to decipher by hand the private key each time we start or reload crowdsec or store the key unencrypted. In any way to strip the passphrase one can do:
openssl rsa -in encrypted-key.pem -out key.pem
Then, the unencrypted key file can be safely deleted after Crowdsec is started.
Configure crowdsec for using a self-signed certificate
server-1 we have to tell crowdsec to use the generated certificate. Hence, the
tls.key_file option in the
api.server section of the following
/etc/crowdec/config.yaml excerpt set to the generated certificate file.
api: server: log_level: info listen_uri: 10.0.0.1:8080 profiles_path: /etc/crowdsec/profiles.yaml online_client: # Crowdsec API credentials (to push signals and receive bad tls: cert_file: /etc/crowdsec/ssl/cert.pem key_file: /etc/crowdsec/ssl/key.pem
On the client side configuration changes happen in two files. First we have to modify
/etc/crowdec/config.yaml to accept self-signed certificates by setting the
We have to change
https in the
/etc/crowdsec/local_api_credentials.yaml file too in order to reflect the changes. This small change has to be done on all three servers (
url: https://10.0.0.1:8080/ login: <login> password: <password>
Side note: Obviously using self-signed certificates doesn't provide any confidence over ownership on the lapi server: servers using the service (
server-3 in our setup) are still vulnerable to man in the middle attack, but at least this setup provides encrypted communications. That's the reason why the
InsecureSkipVerify option is needed.
Using a certificate authority issued certificate
One could say that letsencrypt, or services like Amazon ACM could be leveraged to workaround the
InsecureSkipVerify, by issuing a certificate for a fully qualified domain name that could be added to
/etc/hosts or to a local dns server.
/etc/crowdsec/local_api_credentials.yaml could then be filled with this specified fully qualified domain name.
This indeed works and avoids the
InsecureSkipVerify option to be set. This ensures that communication between client and server can't be tampered with as long as dns configuration can be trusted, but should still be considered as a workaround.
Using a PKI
It's not this blog post's purpose to show how to configure and manage an SSL pki. Please have a look at openssl official openssl documentation. The simple pki scenario is enough to make the Crowdsec stuff work.
Following this documentation, there's a few thing worth mentioning:
To be usable in our crowdsec tls scenario the certificate requests have to be issued with a subject alternative name corresponding to the IP of the Crowdsec LAPI server. This can be done by positioning the
environment variable when invoking openssl for the certificate request (cf step 3.3 in the openssl simple pki scenario):
SAN=IP:10.0.0.1 openssl req -new -config etc/server.conf -out certs/crowdsec.csr -keyout certs/crowdsec.key
The public part of the root and the signing certificates (bundle file created at step 4.5 in the openssl simple pki scenario) have to be added to the local certificate store before starting crowdsec agent. In this setup, this is required to connect to the LAPI server. There're many ways to do so, golang sources specify where certificates are expected, or one can use the
SSL_CERT_FILEenvironment variable in the systemd service file to specify where to find the certificate when launching crowdsec agent.
This article gives some highlights on how to ensure secure communications between our different crowdsec installations. The considered use-case is crowdsec installations in a private network, but this can be deployed as well on a public network with communication over the internet. In such a case a third party certificate would easily do the trick.
Depending on the needs, I proposed three different ways to achieve tls secure communications between crowdsec instances.
The first scenario with self-signed certificates if one only wants to ensure encrypted communication with no need for authentication. The second scenario proposed may only be considered as a workaround when one has the possibility to modify local dns resolutions. The third proposed scenario is the most complicated, but would fit in most use-cases and may be the way to go when security concerns are high.