How to set up a CrowdSec multi-server installation

on April 30, 2021

Introduction

CrowdSec is an open-source & collaborative security solution built to secure Internet-exposed Linux services, servers, containers, or virtual machines with a server-side agent. It is a modernized version of Fail2ban which was a great source of inspiration to the project founders.

CrowdSec is free (under an MIT License) and its source code available on GitHub. The solution is leveraging a log-based IP behavior analysis engine to detect attacks. When the CrowdSec agent detects any aggression, it offers different types of remediation to deal with the IP behind it (access prohibition, captcha, 2FA authentication etc.). The report is curated by the platform and, if legitimate, shared across the CrowdSec community so users can also protect their assets from this IP address.

A few months ago, we added some interesting features to CrowdSec when releasing v1.0.x. One of the most exciting ones is the ability of the CrowdSec agent to act as an HTTP rest API to collect signals from other CrowdSec agents. Thus, it is the responsibility of this special agent to store and share the collected signals. We will call this special agent the LAPI server from now on.

Another worth noting feature, is that mitigation no longer has to take place on the same server as detection. Mitigation is done using bouncers. Bouncers rely on the HTTP REST API served by the LAPI server.

Goals

In this article we’ll describe how to deploy CrowdSec in a multi-server setup with one server sharing signal.

Both server-2 and server-3 are meant to host services. You can take a look on our Hub to know which services CrowdSec can help you secure. Last but not least, server-1 is meant to host the following local services:

the local API needed by bouncers
the database fed by both the three local CrowdSec agents and the online CrowdSec blocklist service. As server-1 is serving the local API, we will call it the LAPI server.

We choose to use a postgresql backend for CrowdSec database in order to allow high availability. This topic will be covered in future posts. If you are ok with no high availability, you can skip step 2.

Furthermore this post will cover attack mitigation for hosted services on server-2 and server-3 using CrowdSec bouncers.

This article is the first of a series.

Prerequisites

Two Internet-facing preinstalled Ubuntu 20.04 servers hosting services. From now on, we will refer to these servers by server-2 and server-3 .
One non Internet facing preinstalled Ubuntu 20.04 server. From now on we will refer to this server by server-1. Let’s assume that server-1's ip is 10.0.0.1. (no internet connection on this server is not a strict requirement).
A local network connecting all three servers

Step 1: CrowdSec installation

Let’s install CrowdSec on every single server, following the CrowdSec installation guide.

wget -qO - https://s3-eu-west-1.amazonaws.com/crowdsec.debian.pragmatic/crowdsec.asc |sudo apt-key add - && echo "deb https://s3-eu-west-1.amazonaws.com/crowdsec.debian.pragmatic/$(lsb_release -cs) $(lsb_release -cs) main" | sudo tee /etc/apt/sources.list.d/crowdsec.list > /dev/null

sudo apt update


sudo apt install crowdsec

We now have three standard CrowdSec installations running.

Step 2 (Optional): Switch the database backend to postgresql on server-1

sudo apt install postgresql

First we have to connect to the database as a postgres user.

sudo -i -u postgres

psql

Thanks to the Postgresql Crowdsec documentation, we can now initialize the database.

postgres=# CREATE DATABASE crowdsec;

CREATE DATABASE

postgres=# CREATE USER crowdsec WITH PASSWORD 'CREATE USER crowdsec WITH PASSWORD '<password>';

postgres=# CREATE USER crowdsec WITH PASSWORD '<password>';

CREATE ROLE

postgres=# GRANT ALL PRIVILEGES ON DATABASE crowdsec TO crowdsec;

GRANT

Now let’s make CrowdSec know about this new database backend. To achieve this, we will have to update the db_config section of the /etc/crowdsec/config.yaml file.

db_config:

  log_level: info

  type: postgres

  user: crowdsec

  password: "<password>"

  db_name: crowdsec

  host: 127.0.0.1

  port: 5432

After registering the local machine again in the database, we are able to restart CrowdSec:

sudo cscli machines add -a

sudo systemctl restart crowdsec

Step 3: Make server-2 and server-3 report to LAPI server

First we have to configure CrowdSec on server-1 to accept connections from server-2 and server-3. Please ensure that your firewall allows connections from server-2 and server-3 on server-1's port 8080.

Let’s configure the API server on server-1 side. For this to happen, we will have to modify both /etc/crowdsec/config.yaml and /etc/crowdsec/local_api_credentials.yaml.

For /etc/crowdsec/config.yaml, it is now the API section that needs to be amended. It’s only a matter of updating the listening IP from localhost to the local IP:

api:

  client:

    insecure_skip_verify: false

    credentials_path: /etc/crowdsec/local_api_credentials.yaml

  server:

    log_level: info

    listen_uri: 10.0.0.1:8080

    profiles_path: /etc/crowdsec/profiles.yaml

    online_client: # Crowdsec API credentials (to push signals and receive bad IPs)                                                                            

      credentials_path: /etc/crowdsec/online_api_credentials.yaml

For /etc/crowdsec/local_api_credentials.yaml we only have to change the configured ip accordingly:

url: http://10.0.0.1:8080/

login: <login>

password: <password>

And we can restart CrowdSec:

sudo systemctl restart crowdsec

Now we will configure the connections on server-2 and server-3.

First we register to the lapi server on both server-2 and server-3:

sudo cscli lapi register -u http://10.0.0.1:8080

By default, the local api server is active on every CrowdSec agent installation. In this setup, we want to disable it on server-2 and server-3. To achieve this, we need to tweak the CrowdSec agent systemd service file.

sudo cp /lib/systemd/system/crowdsec.service /etc/systemd/system/crowdsec.service

Now edit /etc/systemd/system/crowdsec.service and add the -no-api parameter to CrowdSec agent invocation on both server-2 and server-3.

[Unit]

Description=Crowdsec agent

After=syslog.target network.target remote-fs.target nss-lookup.target


[Service]

Type=notify

Environment=LC_ALL=C LANG=C

PIDFile=/var/run/crowdsec.pid

ExecStartPre=/usr/bin/crowdsec -c /etc/crowdsec/config.yaml -t

ExecStart=/usr/bin/crowdsec -c /etc/crowdsec/config.yaml -no-api

#ExecStartPost=/bin/sleep 0.1

ExecReload=/bin/kill -HUP $MAINPID


[Install]

WantedBy=multi-user.target

We can now acknowledge the changes and restart CrowdSec once again.

sudo systemctl daemon-reload

sudo systemctl restart crowdsec

Last thing to do is to allow server-2 and server-3 connections on server-1.

sudo cscli machines list

--------------------------------------------------------------------------------------------------------------------------------------------------------------------

 NAME                                              IP ADDRESS      LAST UPDATE           STATUS  VERSION                                                            

--------------------------------------------------------------------------------------------------------------------------------------------------------------------

 dc6f34b3a4994700a2e333df43728701D0iARTSQ6dxiwyMR  10.0.0.1  2021-04-13T12:16:11Z  ✔️       v1.0.9-4-debian-pragmatic-a8b16a66b110ebe03bb330cda2600226a3a862d7 

 9f3602d1c9244f02b0d6fd2e92933e75zLVg8zSRkyANxHbC  10.0.0.3   2021-04-13T12:24:12Z  🚫                                                                         

 ac86209e6f9c4d7d8de43e2ea31fe28ebvde0vWDr46Mpd3L  10.0.0.2   2021-04-13T12:22:28Z  🚫                                                                         

--------------------------------------------------------------------------------------------------------------------------------------------------------------------

In this output, we can see two machines that are not yet validated. Let’s validate them now.

sudo cscli machines validate 9f3602d1c9244f02b0d6fd2e92933e75zLVg8zSRkyANxHbC

sudo cscli machines validate ac86209e6f9c4d7d8de43e2ea31fe28ebvde0vWDr46Mpd3L

server-2 and server-3 are now allowed to push data to server-1 CrowdSec agent. It may be needed to restart CrowdSec on server-2 and server-3.

sudo systemctl restart crowdsec

On server-1, the command sudo cscli machines list should now show three validated machines.

Step 4: Set up Mitigation

Now we want to install mitigation on our internet-facing servers. First we have to generate two API tokens for server-2 and server-3 on server-1.

sudo cscli bouncers add server-2

Api key for 'server-2':


   02954e85c72cf442a4dee357f0ca5a7c


Please keep this key since you will not be able to retrive it!

sudo cscli bouncers add server-3

Api key for 'server-3':


   3b1030ce0840c343eecd387ac5a3a614


Please keep this key since you will not be able to retrieve it!

For now, there is no package available for the firewall bouncer. This is on our high priority to-do list.

Hence on both server-2 and server-3:

wget https://github.com/crowdsecurity/cs-firewall-bouncer/releases/download/v0.0.10/cs-firewall-bouncer.tgz

tar zxvf cs-firewall-bouncer.tgz

cd cs-firewall-bouncer-v0.0.10/

sudo ./install.sh

If iptables and nftables are not installed, the install script will ask for your permission to install it. The same will happen for ipset installation.

It iss now time to use the token we generated at the beginning of this step. Both api_key and api_url have to be updated in /etc/crowdsec/cs-firewall-bouncer/cs-firewall-bouncer.yaml:

mode: iptables

piddir: /var/run/

update_frequency: 10s

daemonize: true

log_mode: file

log_dir: /var/log/

log_level: info

api_url: http://10.0.0.1:8080/

api_key: 02954e85c72cf442a4dee357f0ca5a7c

disable_ipv6: false

#if present, insert rule in those chains

iptables_chains:

  - INPUT

#  - FORWARD

#  - DOCKER-USER

We can now restart the firewall bouncer.

sudo systemctl restart cs-firewall-bouncer

Conclusion and perspectives

We described how to setup a CrowdSec multi-server installation. The resource overhead on server-2 and server-3 is quite limited as most of the tasks are deported to server-1. This allows to grow the setup by only:

register and validate the CrowdSec agent on the LAPI server
add and validate new bouncers It is worth noting that bouncers and CrowdSec agents don’t have to be installed on the same server. Therefore, the CrowdSec agent has to be installed where logs are generated, but the mitigation can be deported where it is meaningful.

Obviously, there are caveats in this setup:

Communications between agents are over clear http. This is acceptable on a local network, but not possible over the Internet. CrowdSec allows the use of https for those communications, a next post will cover this topic.
Monitoring or alerting is not covered in this article either. CrowdSec allows very powerful monitoring through Prometheus scraper. A post will cover this topic too.
The CrowdSec database is not highly available. Furthermore, the CrowdSec agent on server-1 is a single point of failure.

Now you may be wondering: how to build a highly available multi-machine CrowdSec setup? Stay tuned for our next article.

The CrowdSec team is always happy to receive feedback about the solution and its usage. If you wou like to meet with them and have a chat, here are a few useful links. Give them a shout!

Manuel defended his PhD on Quantum Key Distribution in 2009 and worked as SRE for a few years before becoming involved in security software. He joined CrowdSec early 2021 as a member of its core team and is dedicated to product development 24/7.