If you run your applications in docker containers, use iptables as a firewall and want to block IPs with malicious signs with fail2ban, you’re in the right place.
The setup I struggled with is quite simple:
postfix running in a docker container
logging with the syslog driver into /var/log/mail.log
iptables and fail2ban installed to the host machine
[postfix] jail enabled in fail2ban
With the default settings fail2ban is creating the block rules into iptables’ INPUT chain in the filter table when irregular activity is detected in /var/log/mail.log:
-A fail2ban-postfix -s 184.108.40.206/32 -j REJECT --reject-with icmp-port-unreachable
Despite that the connections from 220.127.116.11 were still hitting my postfix server and I was seeing messages like this in /var/log/fail2ban.log:
Jun 19 12:09:32 localhost fail2ban.actions: INFO [postfix] 18.104.22.168 already banned
The reason for this are the PREROUTING chain rules in the iptables’ nat table that docker creates automatically:
After NATing the packets hit the FORWARD chain in the iptables’ filter table, not INPUT as fail2ban is expecting.
After putting all the pieces together the solution was also quite simple. I created two new files:
From fail2ban manual: The directory action.d contains different scripts defining actions. The actions are executed at well-defined moments during the execution of Fail2ban: when starting/stopping a jail, banning/unbanning a host, etc.
From fail2ban manual: The directory filter.d contains mainly regular expressions which are used to detect break-in attempts, password failures, etc.
The last thing needed was to take the new action and filter into use in fail2ban config file:
That’s it, bans are now appearing correctly into /var/log/fail2ban.log:
2016-04-13 22:18:01,772 fail2ban.actions: WARNING [postfix] Ban 22.214.171.124
2016-04-14 06:22:42,922 fail2ban.actions: WARNING [postfix] Ban 126.96.36.199
The iptables rules are also in order:
-A FORWARD -p tcp -m multiport --dports 25 -j fail2ban-postfix
-A fail2ban-postfix -s 188.8.131.52/32 -j DROP
I still had to solve the problem of loading the iptables rules on boot (with my own custom rules while trying not to break docker), but that’s going to be a separate blog post in the near future.
Prometheus’ alertmanager seems to be very picky on what kind of SMTP certificates it accepts. At the time of writing this post there is no way to tell alertmanager not to use STARTTLS. See issues/193 and pull/266.
I’m running my home baked postfix docker container which is using self signed certificates for the submission port 587. These certificates have to be created so that the postfix server IP is listed in the SAN information. Without this the following error occurs:
time="2016-04-14T08:13:37Z" level=warning msg="Notify attempt 1 failed: starttls failed: x509: cannot validate certificate for 172.17.0.1 because it doesn't contain any IP SANs" source="notify.go:193"
I created the certificates with the following script:
The important part is how the extfile.cnf is used when signing the certificate. I’ve linked alertmanager and postfix containers and binded alertmanager to the docker0 bridge’s IP 172.17.0.1 (since I’m only using the port 587 internally). This IP can be used in the configuration:
[ v3_req ]
subjectAltName = @alt_names
# Extensions to add to a certificate request
basicConstraints = CA:FALSE
keyUsage = nonRepudiation
IP.1 = 172.17.0.1
DNS.1 = amigapallo.org
Now when the certificates are good, they have to be taken into use in alertmanager and postfix. The easiest way that I could come up with is to install the certificates to the host machine’s /etc/ssl/certs and /etc/ssl/private and then link the folders into the containers. I did this with the following Ansible script (but you’ll see easily what’s going on even if you don’t know Ansible):
Here’s what my Ansible script for starting the alertmanager docker container looks like (with non important parts omitted):
Postfix is of course mounting the same directories. If you have authentication in your postfix, you’ll want to set SMTP_AUTH_USERNAME and SMTP_AUTH_PASSWORD environment variables for alertmanager. Another thing you’ll want to configure is the smtp_smarthost in alertmanager.yml:
After all that configuration the alerts are finally being sent:
Lost of time and reading went into this. Here’s a list of sources I found helpful:
Here’s a list of linux commands that I’ve gathered during the years. It seems that every time I want extract a tar.gz or chain couple of commands, I’ve totally forgotten how to do it. I don’t like reading man pages either. A good alternative to this is tldr but I’ve yet to install it. The plan is to keep updating this page whenever I have to Google how to use a certain command. I’m hoping this helps other people too!
Common bash commands
Redirect command stdout and stderr into /dev/null:
Find all shell script files from a certain directory and make them executable:
Repeat a command indefinitely between intervals:
Echo multiple lines into a file:
Show file chmod in numerical format:
Create and extract a tar.gz file:
Match with multiple words using awk, replace the line contents with sed using regex and run a command for each resulting lines:
Replace text in a file with sed:
Set root password without prompt using expect:
Get the directory of the executed script:
Promt user to confirm before continuing:
Conditional variable assignment and a default value:
Validate script parameters and print usage instructions if validation fails:
Do something if a file does not exist:
Do something if the previous command exited with a failure status code:
Curl, wget and scp
Set the host header, user agent and basic auth for a curl request. Follow all redirects and ignore non valid (self signed) certificates:
Set headers and user agent for a wget request and discard the return data:
Make a SOAP request with curl:
Transfer a file to a remote location using scp:
Mail servers, ssl certificates, etc.
Show the certificate information of a mail server:
Telnet to a mail server and send an email:
System management, statistics and monitoring
Get the group and user id from a directory:
Change user and group ids. The command above and this one may become handy when dealing with mounted volumes:
Show the number of connections to a specified port, group them by IP and sort by count:
Monitor incoming packets on a certain port and match against their data payload with a keyword:
Show the total memory and cpu usage for a specified user:
Monitor a log file for 10 seconds and count the lines that match to a certain keyword:
Kill processes matching to a keyword:
Run command as another user using bash as the shell: