Amigapallo A web development blog by Anssi Kinnunen

Configuring alertmanager docker container for a self signed SMTP server certificate

Prometheus’ alertmanager seems to be very picky on what kind of SMTP certificates it accepts. At the time of writing this post there is no way to tell alertmanager not to use STARTTLS. See issues/193 and pull/266.

I’m running my home baked postfix docker container which is using self signed certificates for the submission port 587. These certificates have to be created so that the postfix server IP is listed in the SAN information. Without this the following error occurs:

time="2016-04-14T08:13:37Z" level=warning msg="Notify attempt 1 failed: starttls failed: x509: cannot validate certificate for 172.17.0.1 because it doesn't contain any IP SANs" source="notify.go:193"

I created the certificates with the following script:

CWD="$( cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd )"

# Prepare
rm -rf $CWD/keys
mkdir -p $CWD/keys

# Create a root private key
openssl genrsa -out $CWD/keys/cert.ca.key 2048

# Create a self-signed root certificate
openssl req -x509 -new -nodes -key $CWD/keys/cert.ca.key \
    -days 3650 -out $CWD/keys/postfix.ca.crt \
    -subj "/C=FI/ST=Uusimaa/L=Helsinki/O=Home/CN=amigapallo.org"

# Create a private key for the final certificate
openssl genrsa -out $CWD/keys/postfix.key 2048

# Create a certificate signing request
openssl req -new -key $CWD/keys/postfix.key \
    -out $CWD/keys/cert.csr \
    -subj "/C=FI/ST=Uusimaa/L=Helsinki/O=Home/CN=amigapallo.org"

# Create a server certificate based on the root CA certificate
# and the root private key (and add extensions)
openssl x509 -req -in $CWD/keys/cert.csr \
    -CA $CWD/keys/postfix.ca.crt \
    -CAkey $CWD/keys/cert.ca.key -CAcreateserial \
    -out $CWD/keys/postfix.crt \
    -days 3650 -extensions v3_req -extfile $CWD/extfile.cnf

echo -e "\nUse postfix.ca.crt, postfix.key and postfix.crt\n"

The important part is how the extfile.cnf is used when signing the certificate. I’ve linked alertmanager and postfix containers and binded alertmanager to the docker0 bridge’s IP 172.17.0.1 (since I’m only using the port 587 internally). This IP can be used in the configuration:

[ v3_req ]
subjectAltName = @alt_names

# Extensions to add to a certificate request
basicConstraints = CA:FALSE
keyUsage = nonRepudiation

[alt_names]
IP.1 = 172.17.0.1
DNS.1 = amigapallo.org

Now when the certificates are good, they have to be taken into use in alertmanager and postfix. The easiest way that I could come up with is to install the certificates to the host machine’s /etc/ssl/certs and /etc/ssl/private and then link the folders into the containers. I did this with the following Ansible script (but you’ll see easily what’s going on even if you don’t know Ansible):

- copy: src=postfix.key dest=/etc/ssl/private/postfix.key
- copy: src=postfix.crt dest=/etc/ssl/certs/postfix.crt
- copy: src=postfix.ca.crt dest=/etc/ssl/certs/postfix.ca.crt

- name: install custom certificates
  copy: src= dest=/usr/local/share/ca-certificates/
  with_items:
    - postfix.crt
    - postfix.key
    - postfix.ca.crt

- name: update root certificate database
  command: update-ca-certificates
  register: updated_certs

- debug: msg="{{ updated_certs.stdout }}"

Here’s what my Ansible script for starting the alertmanager docker container looks like (with non important parts omitted):

- name: start alertmanager docker container
  command: 'docker run -d
    --restart=always
    -v "/etc/ssl/certs:/etc/ssl/certs:ro"
    -v "/etc/ssl/private:/etc/ssl/private:ro"
    -v "/usr/local/share/ca-certificates:/usr/local/share/ca-certificates:ro"
    -e SMTP_AUTH_USERNAME=username
    -e SMTP_AUTH_PASSWORD=password
    --name alertmanager
    prom/alertmanager
    -config.file=/alertmanager.yml
'

Postfix is of course mounting the same directories. If you have authentication in your postfix, you’ll want to set SMTP_AUTH_USERNAME and SMTP_AUTH_PASSWORD environment variables for alertmanager. Another thing you’ll want to configure is the smtp_smarthost in alertmanager.yml:

global:
  smtp_from: 'prometheus@amigapallo.org'
  smtp_smarthost: '172.17.0.1:587'

After all that configuration the alerts are finally being sent:

Alert email

Lost of time and reading went into this. Here’s a list of sources I found helpful: