Nginx + Ansible + HTTPS with LetsEncrypt and Nagios on top

Hi folks.
Today I would like demonstrate how to use Ansible in order to construct a server hosting multiple HTTPS domains with Nginx and LetsEncrypt. Are you ready? Let’s dive in.

What you will need

There’s only one thing you will need, and that’s Ansible. If you would like to run local tests without a remote server, then you will need Vagrant and VirtualBox. But those two are optional.

What we are going to set up

The setup is as follows...

Nagios

We are going to have a Nagios with a custom check for pending security updates. That will run under nagios.example.com.

Hugo Website

The main website is going to be a basic Hugo site. Hugo is a static Go based website generator. 
We are also going to setup NoIP which will provide the DNS for the sites.

Wiki

The wiki is a plain, basic DokuWiki.

HTTPS + Nginx

And all the above will be hosted by Nginx with HTTPS provided by letsencrypt. We are going to set all these up with Ansible on top, so it will be idempotent.

Repository

All of the playbooks and the whole thing together can be viewed here: Github Ansible Server Setup.

Ansible

I won’t be writing everything down about Ansible. For that you will need to go and read its documentation. But I will provide ample clarification for using what I’ll be using.

Some basics

Ansible is a configuration management tool which, unlike chef or puppet, isn’t master - slave based. It’s using SSH to run a set of instructions on a target machine. The instructions are written in yaml files and look something like this:

---
# tasks file for ssh
- name: Copy sshd_config
copy: content="{{sshd_config}}" dest=/etc/ssh/sshd_config
notify:
- SSHD Restart

This is a basic Task which copies over an sshd_config file, overwriting the original. It can execute in privileged mode if root password is provided or the user has sudo rights. Ansible works from hosts files where the server details are described. This is how a basic hosts file looks like:

[local]
127.0.0.1

[webserver1]
1.23.4.5

Ansible will use these settings to try and access the server. To test if the connection is working, you can send a ping task to all configured servers like this:

ansible all -m ping

Ansible can also use dynamic entries. But describing those is not part of this post.

Ansible uses variables for things that change. They are defined under each tasks’ subfolder called vars. Please feel free to change the variables to your liking.

SSH Access

You can either define SSH information per host, per group or globally. In this example, I have it under the groups’ vars called webserver1, like this: (vars.yaml):

---
# SSH sudo keys and pass
ansible_become_pass:'{{vault_ansible_become_pass}}'
ansible_ssh_port:'{{vault_ansible_ssh_port}}'
ansible_ssh_user:'{{vault_ansible_ssh_user}}'
ansible_ssh_private_key_file:'{{vault_ansible_ssh_private_key_file}}'
home_dir:/root

Further readings

For further information on Ansible basics, check out the documents below:

• Servers For Hackers
• Ansible docs

Vault

The vault is the place where we can keep secure information. This file is called vault and usually lives under either group_vars or host_vars. The preference is up to you.
This file is encrypted using a password that you’ve specified. You can have the vault password stored via the following ways:

• Store it on a secure drive which is encrypted and only mounted when the playbook is executed;
• Store it on Keybase;
• Store it on an encrypted S3 bucket;
• Store it in a file next to the playbook which is never committed into source control.

Either way, in the end, ansible will look for a file called .vault_password for when it’s trying to decrypt the file. You can define a different file in the ansible.cfg file using the vault_password_file option. You can create a vault like this:

ansible-vault create vault

If you have been following along, you will need the below variables in the vault:

vault_ansible_become_pass:<your_sudo_password> # if applicable

vault_ansible_ssh_user:<ssh_user>

vault_ansible_ssh_private_key_file:/Users/user/.ssh/ida_rsa

vault_nagios_password: supersecurenagiosadminpassword

vault_nagios_username: nagiosadmin

vault_noip_username: youruser@gmail.com

vault_noip_password:"SuperSecureNoIPPassword"

vault_nginx_user:<localuser>

You can always edit the vault later on with:

ansible-vault edit group_vars/webserver1/vault --vault -password-file=.vault_pass

Tasks

The following are a collection of tasks which execute in order. The end task, which is letsencrypt, relies on all the hosts being present and configured under Nginx; otherwise it will throw an error to indicate that the host you are trying to configure HTTPS for isn’t defined.

NO-IP

I’m choosing No-ip as a DNS provider because it’s cheap and the sync tool is easy to automate. To automate the CLI of No-IP, I’m using a package called expect. It will look something like this:

cd {{home_dir}}
wget http://www.no-ip.com/client/linux/noip-duc-linux.tar.gz
mkdir -p noip
tar zxf noip-duc-linux.tar.gz-C noip
cd noip/*
make

/usr/bin/expect <<END_SCRIPT
spawn make install
expect "Please enter the login/email*" { send "{{noip_username}}\r" }
expect "Please enter the password for user*" { send "{{noip_password}}\r" }
expect {
"Do you wish to have them all updated*" {
send "y"
exp_continue
}
}
expect "Please enter an update interval*" { send "30\r" }
expect "Do you wish to run something at successful update*" {send "N" }
END_SCRIPT

The interesting part is the command running expect. Basically, it’s expecting some kind of output which is outlined there, and has canned answers for those which it sends to the waiting command. This is a convenient but fragile way of automating CLI tools. If the expected message changes too much, the automation will no longer work. As a practice, try not to be too specific when matching messages. Expect can use regex.

To Util or Not To Util

There are small tasks- like installing vim, wget and such- which may warrant the existence of a utils task. Utils task may install packages that are used for convenience sake but don’t relate to a single task. The other option would be to let the tasks handle their dependencies.

I settled for the following: Each of my tasks has a dependency part. The given tasks takes care of all the packages it needs so they can be executed on their own as well as in unison. This looks like this:

# Install dependencies
- name: Install dependencies
apt: pkg="{{item}}" state=installed
with_items:
-"{{deps}}"

For which the deps variable is defined as follows:

# Defined dependencies for letsencrypt task.
deps:['git', 'python-dev', 'build-essential', 'libpython-dev', 'libpython2.7', 'augeas-lenses', 'libaugeas0', 'libffi-dev', 'libssl-dev', 'python-virtualenv', 'python3-virtualenv', 'virtualenv']

This is much cleaner. And if a task is no longer needed, it’s dependencies will no longer be needed, ( for most of the cases). That said, I also have a utils task which installs the convenience packages as well. But they aren’t strictly needed and I might not run that task at all.

Nagios

I’m using Nagios 4 which is a real pain to install. Luckily, thanks to Ansible, I only ever had to figure it out once. Now I have a script for that. Installing Nagios demands several, smaller components to be installed. Our task uses import from outside tasks, like this:

- name: Install Nagios
block:
- include_tasks: create_users.yml # creates the Nagios user
- include_tasks: install_dependencies.yml # installs Nagios dependencies
- include_tasks: core_install.yml # Installs Nagios Core
- include_tasks: plugin_install.yml # Installs Nagios Plugins
- include_tasks: create_htpasswd.yml # Creates a password for Nagios' admin user
- include_tasks: setup_custom_check.yml # Adds a custom check which is to check how many security updates are pending
when: st.stat.exists == False

The when is a check for a variable created by a file check.

- stat:
path:/usr/local/nagios/bin/nagios
register: st

It checks whether Nagios has been installed or not. If yes, then skip. I’m not going to paste all the subtasks here because that would be too huge. You can check the subtasks out in the repository under Nagios.

Hugo

Hugo is easy to install. Its sole requirement is Go. To install hugo, simply run apt-get install hugo. For setting up the site, check out a sample from hugo’s website and than execute hugo from the root folder, like this:

hugo server --bind=127.0.0.1 --port=8080 --baseUrl=https://example.com --appendPort=false --logFile hugo.log --verboseLog --verbose -v &

Wiki

I used DokuWiki because it’s a file based wiki. Installation is basically just downloading the archive, extracting it and- done! The only thing that’s needed for it, is php-fpm to serve it and a few php modules, (which I’ll outline in the ansible playbook).

The VHOST file for DokuWiki is provided by them and looks like this:

server {
server_name {{ wiki_server_name }};
root {{ wiki_root }};
index index.php index.html index.htm;
client_max_body_size 2M;
client_body_buffer_size 128k;
location /{
index doku.php;
try_files $uri $uri/ @dokuwiki;
}
location @dokuwiki {
rewrite ^/_media/(.*)/lib/exe/fetch.php?media=$1 last;
rewrite ^/_detail/(.*)/lib/exe/detail.php?media=$1 last;
rewrite ^/_export/([^/]+)/(.*)/doku.php?do=export_$1&id=$2 last;
rewrite ^/(.*)/doku.php?id=$1 last;
}
location ~ \.php$ {
try_files $uri =404;
fastcgi_pass unix:/var/run/php5-fpm.sock;
fastcgi_index index.php;
fastcgi_param SCRIPT_FILENAME $document_root$fastcgi_script_name;
include fastcgi_params;
}
location ~ /\.ht{
deny all;
}
location ~ /(data|conf|bin|inc)/{
deny all;
}
}

Nginx

Nginx install is through apt as well. Here, however, there is a bit of magic going on with templates. The templates provide the vhost files for the three hosts we will be running. It will look like the following:

- name: Install vhosts
block:
- template: src=01_example.com.j2 dest=/etc/nginx/vhosts/01_example.com
notify:
- Restart Nginx
- template: src=02_wiki.example.com.j2 dest=/etc/nginx/vhosts/02_wiki_example.com
notify:
- Restart Nginx
- template: src=03_nagios.example.com.j2 dest=/etc/nginx/vhosts/03_nagios.example.com
notify:
- Restart Nginx

Now, you might be wondering what notify is? It’s basically a handler that gets notified to restart nginx. The great thing about it is that it does this only once, even if it was called multiple times. The handler will look like this:

- name: Restart Nginx
service:
name: nginx
state: restarted

It lives under the handlers’ sub-folder. With this, Nginx is done and should convert our sites under plain HTTP. HTTPS is added after this step.

LetsEncrypt

Now comes the part where we enable HTTPS for all these three domains. Which is as follows:

• example.com;
• wiki.example.com;
• nagios.example.com.

This is actually quite simple now-a-days with certbot-auto. In fact, it will insert the configurations we need all by itself. The only thing for us to do now is to specify what domains we have and what our challenge will be. Also, we have to pass in some variables for certbot-auto to run in a non-interactive mode. This will look like the following:

- name: Generate Certificatefor Domains
shell: ./certbot-auto --authenticator standalone --installer nginx -d '{{ domain_example }}'-d '{{ domain_wiki }}'-d '{{ domain_nagios }}'--email example@gmail.com--agree-tos -n --no-verify-ssl --pre-hook "sudo systemctl stop nginx"--post-hook "sudo systemctl start nginx"--redirect
args:
chdir:/opt/letsencrypt

And that’s that! The interesting and required part here is the pre-hook and post-hook. Without these it won’t work. This is because the ports that certbot is performing the challenge on would be already taken. This might not be true or even ideal if you can’t afford to stop nginx. This stops nginx, performs the challenge, generates the certs and starts nginx again. Please note: --redirect. This will force HTTPS on the sites, and will disable plain HTTP.
If all goes well our sites should contain information such as this:

listen 443 ssl; # managed by Certbot
ssl_certificate /etc/letsencrypt/live/example.com-0001/fullchain.pem; # managed by Certbot
ssl_certificate_key /etc/letsencrypt/live/example.com-0001/privkey.pem; # managed by Certbot
include /etc/letsencrypt/options-ssl-nginx.conf; # managed by Certbot
ssl_dhparam /etc/letsencrypt/ssl-dhparams.pem; # managed by Certbot

server {
if($host = example.com){
return301 https://$host$request_uri;
} # managed by Certbot
server_name example.com;
listen 80;
return404; # managed by Certbot
}

Test run using Vagrant

If you don’t want to run all of this on a live server in order to test it out, you can either do these two things:

• Use a remote dedicated test server;
• Use a local virtual machine with Vagrant.

Here, I’m providing you with an option for the latter.
It’s possible for most things to be tested on a local Vagrant machine. For installing things, a Vagrant box is usually enough to test it out. A sample Vagrantfile looks like this:

# encoding: utf-8
# -*- mode: ruby -*-
# vi: set ft=ruby :
# Box/ OS
VAGRANT_BOX ='ubuntu/xenial64'

VM_NAME ='ansible-practice'

Vagrant.configure(2)do|config|
# Vagrant box from Hashicorp
config.vm.box= VAGRANT_BOX
# Actual machine name
config.vm.hostname= VM_NAME
# Set VM name in Virtualbox
config.vm.provider'virtualbox'do|v|
v.name= VM_NAME
v.memory=2048
end
# Ansible provision
config.vm.provision'ansible_local'do|ansible|
ansible.limit='all'
ansible.inventory_path='hosts'
ansible.playbook='local.yml'
end
end

The most interesting part here is the ansible provision section. It’s running a version of Ansible that is called ansible_local. It’s local because it will be only on the VirtualBox, (meaning you don’t have to have Ansible installed to test it on a vagrant box). Neat, huh?

To test your playbook, simply run vagrant up and you should see the provisions happen. If you would like to test whether your websites are available from outside of the virtual box, that will require some port forwarding in VirtualBox and in Vagrantfile.For more on that, please read these two sections:

• https://www.vagrantup.com/docs/networking/forwarded_ports.html
• https://www.vagrantup.com/docs/networking/basic_usage.html

Room for improvement

And that’s it. Please note that this setup isn’t quite enterprise ready. I would add the following things:

Tests and Checks

A ton of tests and checks to see whether the commands that we are using are actually successful or not. If they aren’t, make them report the failure.

Multiple Domains

If you happen to have a ton of domain names that need to be set up, this will not be the most effective way. Right now letsencrypt creates a single certificate file for those three domains with -d, (that’s not what you’ll want with potentially hundreds of domains).

In that case, have a list to go through with with_items. Note that you’ll have to restart nginx on each line because you don’t want one of them fail and stop the process entirely. Rather, have a few fail but the rest still working. This should be okay, since you wouldn’t run a playbook multiple times, just once when you set up a server from scratch.

Conclusion

That’s it folks. Have fun setting up servers all over the place and enjoy the power of nginx and letsencrypt and not having to worry about adding another server into the bunch.

Thank you for reading,
Gergely.