Configuring alertmanager docker container for a self signed SMTP server certificate14 Apr 2016
Prometheus’ alertmanager seems to be very picky on what kind of SMTP certificates it accepts. At the time of writing this post there is no way to tell alertmanager not to use STARTTLS. See issues/193 and pull/266.
I’m running my home baked postfix docker container which is using self signed certificates for the submission port 587. These certificates have to be created so that the postfix server IP is listed in the SAN information. Without this the following error occurs:
time="2016-04-14T08:13:37Z" level=warning msg="Notify attempt 1 failed: starttls failed: x509: cannot validate certificate for 172.17.0.1 because it doesn't contain any IP SANs" source="notify.go:193"
I created the certificates with the following script:
The important part is how the extfile.cnf is used when signing the certificate. I’ve linked alertmanager and postfix containers and binded alertmanager to the docker0 bridge’s IP 172.17.0.1 (since I’m only using the port 587 internally). This IP can be used in the configuration:
[ v3_req ] subjectAltName = @alt_names # Extensions to add to a certificate request basicConstraints = CA:FALSE keyUsage = nonRepudiation [alt_names] IP.1 = 172.17.0.1 DNS.1 = amigapallo.org
Now when the certificates are good, they have to be taken into use in alertmanager and postfix. The easiest way that I could come up with is to install the certificates to the host machine’s
/etc/ssl/private and then link the folders into the containers. I did this with the following Ansible script (but you’ll see easily what’s going on even if you don’t know Ansible):
Here’s what my Ansible script for starting the alertmanager docker container looks like (with non important parts omitted):
Postfix is of course mounting the same directories. If you have authentication in your postfix, you’ll want to set
SMTP_AUTH_PASSWORD environment variables for alertmanager. Another thing you’ll want to configure is the
smtp_smarthost in alertmanager.yml:
After all that configuration the alerts are finally being sent:
Lost of time and reading went into this. Here’s a list of sources I found helpful: