Amigapallo A web development blog by Anssi Kinnunen

Configuring fail2ban and iptables to get along with docker

If you run your applications in docker containers, use iptables as a firewall and want to block IPs with malicious signs with fail2ban, you’re in the right place.

The problem

The setup I struggled with is quite simple:

  • postfix running in a docker container
    • logging with the syslog driver into /var/log/mail.log
  • iptables and fail2ban installed to the host machine
    • [postfix] jail enabled in fail2ban

With the default settings fail2ban is creating the block rules into iptables’ INPUT chain in the filter table when irregular activity is detected in /var/log/mail.log:

-A fail2ban-postfix -s 1.2.3.4/32 -j REJECT --reject-with icmp-port-unreachable

Despite that the connections from 1.2.3.4 were still hitting my postfix server and I was seeing messages like this in /var/log/fail2ban.log:

Jun 19 12:09:32 localhost fail2ban.actions: INFO   [postfix] 1.2.3.4 already banned

The reason for this are the PREROUTING chain rules in the iptables’ nat table that docker creates automatically:

# Jump into DOCKER chain
-A PREROUTING -m addrtype --dst-type LOCAL -j DOCKER

# NAT traffic going to port 25
-A DOCKER ! -i docker0 -p tcp -m tcp --dport 25
    -j DNAT --to-destination 172.17.0.2:25

After NATing the packets hit the FORWARD chain in the iptables’ filter table, not INPUT as fail2ban is expecting.

The solution

After putting all the pieces together the solution was also quite simple. I created two new files:

fail2ban-postfix-action.conf

# /etc/fail2ban/action.d/fail2ban-postfix-action.conf

[Definition]

actionstart = iptables -N fail2ban-postfix
              iptables -A fail2ban-postfix -j RETURN
              iptables -I FORWARD -p tcp -m multiport --dports 25 -j fail2ban-postfix

actionstop = iptables -D FORWARD -p tcp -m multiport --dports 25 -j fail2ban-postfix
             iptables -F fail2ban-postfix
             iptables -X fail2ban-postfix

actioncheck = iptables -n -L FORWARD | grep -q 'fail2ban-postfix[ \t]'

actionban = iptables -I fail2ban-postfix 1 -s <ip> -j DROP

actionunban = iptables -D fail2ban-postfix -s <ip> -j DROP

From fail2ban manual: The directory action.d contains different scripts defining actions. The actions are executed at well-defined moments during the execution of Fail2ban: when starting/stopping a jail, banning/unbanning a host, etc.

fail2ban-postfix-filter.conf.j2

# /etc/fail2ban/filter.d/fail2ban-postfix-filter.conf

[INCLUDES]
before = common.conf

[Definition]

_daemon = postfix/smtpd

# Note that the last 3 regex's are not present by default. The 4th one
# is the most important because the default ones do not match any of the
# lines the docker and the syslog driver are generating.
failregex = ^%(__prefix_line)sNOQUEUE: reject: RCPT from \S+\[<HOST>\]: 554 5\.7\.1 .*$
            ^%(__prefix_line)sNOQUEUE: reject: RCPT from \S+\[<HOST>\]: 450 4\.7\.1 : Helo command rejected: Host not found; from=<> to=<> proto=ESMTP helo= *$
            ^%(__prefix_line)sNOQUEUE: reject: VRFY from \S+\[<HOST>\]: 550 5\.1\.1 .*$
            ^.* postfix.* NOQUEUE: reject: RCPT from \S+\[<HOST>\]: 554 5\.7\.1 .*$
            ^.* postfix.* too many errors after AUTH from \S+\[<HOST>\]$
            ^.* postfix.* warning: \S+\[<HOST>\]: SASL LOGIN authentication failed: authentication failure$

ignoreregex =

From fail2ban manual: The directory filter.d contains mainly regular expressions which are used to detect break-in attempts, password failures, etc.

The last thing needed was to take the new action and filter into use in fail2ban config file:

[postfix]

enabled  = true
port     = smtp
filter   = fail2ban-postfix-filter
logpath  = /var/log/mail.log
banaction = fail2ban-postfix-action

That’s it, bans are now appearing correctly into /var/log/fail2ban.log:

2016-04-13 22:18:01,772 fail2ban.actions: WARNING [postfix] Ban 1.2.3.4
2016-04-14 06:22:42,922 fail2ban.actions: WARNING [postfix] Ban 5.6.7.8

The iptables rules are also in order:

-A FORWARD -p tcp -m multiport --dports 25 -j fail2ban-postfix
-A fail2ban-postfix -s 1.2.3.4/32 -j DROP

I still had to solve the problem of loading the iptables rules on boot (with my own custom rules while trying not to break docker), but that’s going to be a separate blog post in the near future.

Configuring alertmanager docker container for a self signed SMTP server certificate

Prometheus’ alertmanager seems to be very picky on what kind of SMTP certificates it accepts. At the time of writing this post there is no way to tell alertmanager not to use STARTTLS. See issues/193 and pull/266.

I’m running my home baked postfix docker container which is using self signed certificates for the submission port 587. These certificates have to be created so that the postfix server IP is listed in the SAN information. Without this the following error occurs:

time="2016-04-14T08:13:37Z" level=warning msg="Notify attempt 1 failed: starttls failed: x509: cannot validate certificate for 172.17.0.1 because it doesn't contain any IP SANs" source="notify.go:193"

I created the certificates with the following script:

CWD="$( cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd )"

# Prepare
rm -rf $CWD/keys
mkdir -p $CWD/keys

# Create a root private key
openssl genrsa -out $CWD/keys/cert.ca.key 2048

# Create a self-signed root certificate
openssl req -x509 -new -nodes -key $CWD/keys/cert.ca.key \
    -days 3650 -out $CWD/keys/postfix.ca.crt \
    -subj "/C=FI/ST=Uusimaa/L=Helsinki/O=Home/CN=amigapallo.org"

# Create a private key for the final certificate
openssl genrsa -out $CWD/keys/postfix.key 2048

# Create a certificate signing request
openssl req -new -key $CWD/keys/postfix.key \
    -out $CWD/keys/cert.csr \
    -subj "/C=FI/ST=Uusimaa/L=Helsinki/O=Home/CN=amigapallo.org"

# Create a server certificate based on the root CA certificate
# and the root private key (and add extensions)
openssl x509 -req -in $CWD/keys/cert.csr \
    -CA $CWD/keys/postfix.ca.crt \
    -CAkey $CWD/keys/cert.ca.key -CAcreateserial \
    -out $CWD/keys/postfix.crt \
    -days 3650 -extensions v3_req -extfile $CWD/extfile.cnf

echo -e "\nUse postfix.ca.crt, postfix.key and postfix.crt\n"

The important part is how the extfile.cnf is used when signing the certificate. I’ve linked alertmanager and postfix containers and binded alertmanager to the docker0 bridge’s IP 172.17.0.1 (since I’m only using the port 587 internally). This IP can be used in the configuration:

[ v3_req ]
subjectAltName = @alt_names

# Extensions to add to a certificate request
basicConstraints = CA:FALSE
keyUsage = nonRepudiation

[alt_names]
IP.1 = 172.17.0.1
DNS.1 = amigapallo.org

Now when the certificates are good, they have to be taken into use in alertmanager and postfix. The easiest way that I could come up with is to install the certificates to the host machine’s /etc/ssl/certs and /etc/ssl/private and then link the folders into the containers. I did this with the following Ansible script (but you’ll see easily what’s going on even if you don’t know Ansible):

- copy: src=postfix.key dest=/etc/ssl/private/postfix.key
- copy: src=postfix.crt dest=/etc/ssl/certs/postfix.crt
- copy: src=postfix.ca.crt dest=/etc/ssl/certs/postfix.ca.crt

- name: install custom certificates
  copy: src= dest=/usr/local/share/ca-certificates/
  with_items:
    - postfix.crt
    - postfix.key
    - postfix.ca.crt

- name: update root certificate database
  command: update-ca-certificates
  register: updated_certs

- debug: msg="{{ updated_certs.stdout }}"

Here’s what my Ansible script for starting the alertmanager docker container looks like (with non important parts omitted):

- name: start alertmanager docker container
  command: 'docker run -d
    --restart=always
    -v "/etc/ssl/certs:/etc/ssl/certs:ro"
    -v "/etc/ssl/private:/etc/ssl/private:ro"
    -v "/usr/local/share/ca-certificates:/usr/local/share/ca-certificates:ro"
    -e SMTP_AUTH_USERNAME=username
    -e SMTP_AUTH_PASSWORD=password
    --name alertmanager
    prom/alertmanager
    -config.file=/alertmanager.yml
'

Postfix is of course mounting the same directories. If you have authentication in your postfix, you’ll want to set SMTP_AUTH_USERNAME and SMTP_AUTH_PASSWORD environment variables for alertmanager. Another thing you’ll want to configure is the smtp_smarthost in alertmanager.yml:

global:
  smtp_from: 'prometheus@amigapallo.org'
  smtp_smarthost: '172.17.0.1:587'

After all that configuration the alerts are finally being sent:

Alert email

Lost of time and reading went into this. Here’s a list of sources I found helpful:

Collection of useful linux commands

Here’s a list of linux commands that I’ve gathered during the years. It seems that every time I want extract a tar.gz or chain couple of commands, I’ve totally forgotten how to do it. I don’t like reading man pages either. A good alternative to this is tldr but I’ve yet to install it. The plan is to keep updating this page whenever I have to Google how to use a certain command. I’m hoping this helps other people too!

Common bash commands

Redirect command stdout and stderr into /dev/null:

cat /non/existing/file &> /dev/null
non-existing-command > /dev/null 2>&1

Find all shell script files from a certain directory and make them executable:

find /etc -name "*.sh" -exec chmod +x {} \;

Repeat a command indefinitely between intervals:

while x=0; do \
    free -m | grep -e '^-' \
        | awk '{ print "Free memory: "$4 }'; \
    sleep 1; \
done

Echo multiple lines into a file:

cat > /tmp/lines.log << EOL
line 1
line 2
EOL

Show file chmod in numerical format:

stat --format '%a' /etc/sysctl.conf

Create and extract a tar.gz file:

# Create
tar -zcvf dump.sql.tar.gz dump.sql

# Extract
tar -zxvf dump.sql.tar.gz

Match with multiple words using awk, replace the line contents with sed using regex and run a command for each resulting lines:

cat /var/log/syslog \
    | awk '/error/ && /TLS/ {print $0}' \
    | sed -r 's/^.*number\s([0-9]+)\:$/\1/' \
    | while read line; do \
        echo "Error number: $line"; \
    done

# awk can be used to mach multiple columns also:
route -n | awk '{ if ($8 == "eth0" && $2 != "0.0.0.0") print $2; }'

Replace text in a file with sed:

sed -i s/#retry_files_enabled/retry_files_enabled/ \
    /etc/ansible/ansible.cfg

Set root password without prompt using expect:

#!/usr/bin/expect
spawn sudo -u root passwd
expect "Enter new UNIX password:"
send "root\r"
expect "Retype new UNIX password:"
send "root\r"
interact

Bash scripts

Get the directory of the executed script:

CWD="$( cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd )"

Promt user to confirm before continuing:

read -p "confirm? " -n 1 -r

if [[ $REPLY =~ ^[Yy]$ ]]; then
    echo "Confirmed!"
fi

Conditional variable assignment and a default value:

env="development"
domain="blog.amigapallo.dev" \
    && [[ $env == "production" ]] && \
    domain="blog.amigapallo.org"

Validate script parameters and print usage instructions if validation fails:

[[ "$1" =~ [production|development] ]] \
    || { echo "Usage: build.sh development|production" >&2; exit 1; }

Do something if a file does not exist:

if [ ! -f /usr/bin/ansible ]; then
    echo "Does not exist!"
fi

Do something if the previous command exited with a failure status code:

if [ $? -ne 0 ]; then
    echo "Failure!"
else
    echo "Success!"
fi

Curl, wget and scp

Set the host header, user agent and basic auth for a curl request. Follow all redirects and ignore non valid (self signed) certificates:

curl --verbose --header 'Host: blog.amigapallo.org' \
    --header "User-Agent: Go-http-client" \
    -u username:password \
    -L \
    --insecure \
    'https://88.198.199.58:80'

Set headers and user agent for a wget request and discard the return data:

wget -O /dev/null http://google.com --server-response --user-agent="Go-http-client" --header="X-Forwarded-For: 172.217.1.206"

Make a SOAP request with curl:

echo "<S:Envelope xmlns:S="http://schemas.xmlsoap.org/soap/envelope/"><S:Body><ns2:getStuff xmlns:ns2="http://ejb.domain.com"><arg0>01110279</arg0><arg1>en</arg1></ns2:getStuff></S:Body></S:Envelope>" > soap.txt

curl --header "Content-Type: text/xml;charset=UTF-8" --data @soap.txt http://domain.com/get

Transfer a file to a remote location using scp:

scp dump.sql.tar.gz user@1.2.3.4:/home/user/dump.sql.tar.gz

Mail servers, ssl certificates, etc.

Show the certificate information of a mail server:

openssl s_client -starttls smtp -crlf -connect smtp.gmail.com:587

Telnet to a mail server and send an email:

# Copy paste the following commands line by line:
telnet 1.2.3.4 25
EHLO domain.com
MAIL FROM: sender@domain.com
RCPT TO: receiver@domain.com
DATA
Subject: Subject!
Body of the message!
.
quit

System management, statistics and monitoring

Get the group and user id from a directory:

GID=$(stat -c "%g" /home/amigapallo)
UID=$(stat -c "%u" /home/amigapallo)

Change user and group ids. The command above and this one may become handy when dealing with mounted volumes:

usermod -u $UID ftp
groupmod -g $GID ftp

# Note that when changing the user and group ids, the existing
# file and directory UIDs and GIDs should be changed also:

find / -gid $OLD_GID ! -type l -not -path '/proc/*' \
    -exec chgrp -h $GID {} \;

find / -user $OLD_UID -not -path '/proc/*' \
    -exec chown -h $UID {} \;

Show the number of connections to a specified port, group them by IP and sort by count:

netstat -plan | grep :80 | grep -v "::" | awk {'print $5'} \
    | cut -d: -f 1 | sort |uniq -c | sort -nk 1

Monitor incoming packets on a certain port and match against their data payload with a keyword:

ngrep -d any port 9052 | grep -i "GET"

Show the total memory and cpu usage for a specified user:

# rss = resident set size, vsz = virtual memory size
echo "rss(KiB) vmem(KiB)"; ps -U mysql --no-headers -o rss,vsz \
    | awk '{rss+=$1; vmem+=$2} END{print rss" "vmem}' | sort -k3

echo "%CPU"; ps -U mysql --no-headers -o %cpu \
    | awk '{cpu+=$1} END{print cpu}' | sort -k2

Monitor a log file for 10 seconds and count the lines that match to a certain keyword:

tail -n0 -f /opt/webserver/logs/access.log | grep "GET" > lines.log \
    & sleep 10; kill $! ; wc -l lines.log; rm lines.log

Kill processes matching to a keyword:

kill -9 `ps -ef | grep docker | grep -v grep | awk '{print $2}'`

Run command as another user using bash as the shell:

su -s /bin/bash mysql -c "env"

Switch to another user and use bash as the shell:

su -s /bin/bash jenkins

Get user and group name by id:

getent group 0 | cut -d: -f1
getent passwd 0 | cut -d: -f1

Managing rpm packages:

# List all installed packages matching to a keyword
rpm -qa | grep epel

# Install rpm package
rpm -ivh epel-release-6-7.noarch.rpm

# Remove (erase) rpm package
rpm -e epel-release-6-7.noarch