Setting up Docker Swarm in AWS with EFS backed storage and Traefik with Consul

Docker is awesome, but Docker is also hard.  The documentation can be sparse or cryptic, and every guide I find seems to only apply for that particular use case, or in testing, or etc.  There is always a footnote stating that more needs to be done.  And use cases can be varied and complicated, but I had a simple task: I needed to have docker containers running in the cloud delivering websites, preferably all in the same swarm to keep costs down and management simple.

This is an attempt to document an easy, dynamic setup for Docker Swarm in AWS that allows a user to have multiple WEBSITES (not just microservices) living in the swarm, with their own microservices if needed, all with SSL certificates auto generated and living behind a load balancer.

In the past few months I have probably spent hundreds of hours trying to figure out the Docker ecosystem and technologies.  I was part of an initiative at my company to move to containerized applications with Docker.  I wanted to move to shipping environments instead of code, and if they are scalable, all the better.

Setting up ad using a swarm requires some re-thinking about how your application will work and how to leverage some technology you may have taken for granted – like volumes and networks.

This will be a quick guide to setting up a swarm in AWS with all the necessary bells and whistles for fast deployment.

Requirements:

  • Scalable architecture
  • One Docker Swarm
  • HTTP to HTTPS redirection and SSL generation for many actual websites (could be up to a few hundred)
  • Load balancing
  • Reverse proxying for multiple TLDs and subdomains, so I can point whatever domains I want at the swarm and it works, as long as there is a service listening for that domain.
  • Persistent storage for SSL certs in a high-availability system – so if I lose my proxy, I can rebuild it in seconds and re-attach to that storage and not have to regenerate certificates
  • Internal sevice discovery and service to service communication – so I can have microservices backing a main website.  The swarm does this automatically.

Step 1:

There are a bazillion ways to create a swarm in AWS, but the most straightforward way is to use the Cloud Formation template Docker has put together.  They have a page with hot links that will take you to your Cloud Formation page in AWS and pre-fill the template for a swarm.

Make sure you have a key-pair for the zone you are working in with AWS so you can SSH to your instances/nodes.

Select the link that works for your needs, probably CE Stable.  You will be taken to the AWS stack formation page with the template pre-selected.  Continuing to the next page will show you options for this template.

You get a few options to determine how many managers and workers will initially be created in the swarm.  I just chose 3 managers and 3 workers.  Also, and this is important, you must enable EFS storage at this point so you can create volumes that are shareable among several running containers – basically, a shared file system.  You can go back later and do this, but it will essentially destroy and rebuild your entire swarm.

This template will create and entire Virtual Private Cloud with all the necessary instances, a load balancer that magically knows about published ports in the swarm (AWESOME), security groups, and the like.

NOTE: There is some confusing terminology, since a ‘Stack’ in AWS is a set of instances, load balancers, subnets, security groups, etc that make up a VPC (Virtual Private Cloud).  A stack in docker is a collection of services running together in a scalable manner.

WARNING: In order to get EFS backed storage so you can share volumes between several containers/services/stacks, you MUST manually enable EFS storage on the setup page for this Cloud Formation Stack.

WARNING: You CANNOT use Docker Cloud to create your swarm if you want EFS backed storage, since it does not enable it by default.

Step 2:

The VPC and Swarm are now running.  Now it is time to set up a connection to your swarm so you can run commands on it.  You have a few different ways to do this.

Method 1: SSH

Using SSH to one of your manager nodes, you can run docker commands all day long.  Unfortunately, this means that you need to copy any files you want Docker to use onto that manager node, which can get messy.  For example, deploying a stack would mean you need to copy that .yml file to that node and then run docker stack deploy from there.

Method 2: SSH Tunneling

This methods involves creating an SSH tunnel to your manager node and then using the export DOCKER_HOST method to send your local Docker commands to the tunnel.

Normally, Docker commands are sent to the local docker socket of your machine.  However, you can set the DOCKER_HOST env variable so that docker sends commands to a different host, either a VM or even a remote machine (like we want to here).  Instead of exporting the env variable, which overwrites it for your current terminal session, you can specify the host target using the -H argument of docker, e.g.:

docker -H <machine>:<port> <command>

The benefit is that you run commands from your machine and send them to the swarm, and also use local stack files to configure and deploy stacks on the remote swarm.  You don’t need to copy the files to the remote, since the command is parsed locally and then sent to the swarm!

For example, you can set up a tunnel:
ssh -i key.pem -NL localhost:2376:/var/run/docker.sock docker@<ip_address_of_manager> &

and then run
export DOCKER_HOST=localhost:2376

Then, for this session on the command line, all docker commands will be sent to this tunnel.  Optionally, you can omit this step and simply use:

docker -H localhost:2376 <commands>

Which accomplishes the same thing.

The downside to this method is the need to maintain your tunnels, which can often break if unused and get weird when moving networks and VPNs.  Therefore, I am not a fan of this method.

Method 3: Exposing the Docker Port

You can also open the Docker Port in your security group for your managers (you will see an appropriately labeled security group for your swarm manager in AWS) and then export the manager IP and port to your DOCKER_HOST env variable.  You will have to use a non-secure connection to do so without setting up some advanced certificates and the like, which is beyond scope here.  The non secure port is 2375 – open that up to your IP pool or VPN (NOT THE WORLD), and then run

export DOCKER_HOST=<manager_node_ip>:2375

Now your docker commands will be sent to that manager.  I do not recommend this setup, since it can be a security risk and does not establish secure connections, and it means if you don’t have a VPN and are on the move, you would have to keep updating the allowed IPs.

Method 3: My Favorite – Docker Cloud

Method 3 involved leveraging Docker Cloud to proxy your commands.  You get the benefits of the SSH tunnel, namely, running commands from your local machine with access to local stack file, without needing to maintain SSH tunnels yourself.  It is also secure, since it uses the secure connection on port 2376 to talk to the docker daemon in the swarm from Docker Cloud.

Log in to Docker Cloud and switch to swarm mode.  Navigate to the swarms page and hit ‘Bring your own swarm’ – a popup will appear with instructions to run a docker command on one of your manager nodes.  Copy that command, SSH in to one of your manager nodes, paste and run.  This command will create a client proxy container that will essentially receive remote docker commands and pipe them into the swarm.  You will be asked to log in to Docker Cloud and set a name for the swarm.

Once that is complete and the swarm is connected to Docker Cloud, your swarm will appear in Docker Cloud with a blue dot, indicating is is running.  You can click your swarm and get a script to run on your local machine that will install another proxy, binding to one of your local ports.  Copy/paste and run it.  Once this is running, you now have a proxy tunnel to your swarm that is managed securely by Docker Cloud.  It will display instructions on what local port this tunnel is connected to, and how to set up your session to use it.  You can export that DOCKER_HOST like in method 2, or just add the -H argument to your docker commands, but either way you now have a stable and manageable connection.  If you forget what port you need to use, just run

docker ps

On your local machine, and you will see the client proxy container and its local port!

Step 3:

You now have a VPC and active Swarm, and a reliable way to communicate to that swarm. Next, we need to set up Traefik as a reverse proxy.   Traefik works by mapping a front end service (a domain name) to a back end service (a service in the swarm).  It does this by reading labels you set on your docker stacks which tell Traefik what domain they want to be associated with, and what port they want to talk on.

Essentially, we want to have ports 80 and 443 open to the world, which would work fine if we had just one service on those ports (the nature of Docker and swarm is that only one service can use a port at a time) – but we want multiple websites.  The reverse proxy handles this by listening to all requests and sending them to a service on the backside using the requested domain/path as a routing mechanism – a lot like how Apache virtual hosts work.  Traefik is awesome because it also handles auto HTTPS redirection and automatically generating SSL certs for each domain.  We store the certificates in a key value store – Consul, per Traefik docs – so we have constant, controlled, and persistent access to these certs, even when Traefik is running in high-availability mode across 3 replicas on 3 nodes.

First, we need to create a network that both Traefik and all our services will use to talk to each other.  If we didn’t specify a network, then every time we ran docker stack deploy, it would create a private network just for that stack and not be able to talk to or know about other stacks, services, or containers.

docker network create -d overlay <network name>

Now let’s create an EFS-backed shared volume for Consul to use.  We need Consul to act as a controller for setting and getting data from a shared source, and it has a Key-Value store that our reverse proxy, Traefik, can use to store configuration and SSL certificates.  We then map an EFS-backed shared volume into Consul so that if Consul dies, we still have all our key value storage and can simply bring up a new Consul service and reconnect with little downtime, and no need to rebuild configurations and regenerate SSL certs.  EFS is our driver mechanism because we can then create volumes that are shareable across multiple services and containers that have been scaled, whereas normal volumes cannot do this.

docker volume create -d "cloudstor:aws" --opt backing=shared {volume_name}

Now we are ready to actually boot up Traefik and Consul.

Step 4:

We want Traefik to be in high-availablity mode, which basically means it will be living on all points of contacts to the swarm – the manager nodes.  The load balancer generated in step 1 listens to all docker events in the That means a request will be routed from the load balancer to a node manager to Traefik, and then to a service on the backend.  To do this, we need to deploy it as a stack, or set of replicated services across all manager nodes.

We actually created a DevOps repo that contained this documentation and our stack file for Traefik and Consul, and then deployed the stack from a local machine using the above proxy method to send that command to the swarm.

This stack file was taken from the Traefik documentation directly, and you can read more about it here.  This is just a quick start guide:

version: "3.4"
services:
  traefik_init:
    image: traefik:1.5
    command:
      - "storeconfig"
      - "--api"
      - "--entrypoints=Name:http Address::80 Redirect.EntryPoint:https"
      - "--entrypoints=Name:https Address::443 TLS"
      - "--defaultentrypoints=http,https"
      - "--acme"
      - "--acme.storage=traefik/acme/account"
      - "--acme.entryPoint=https"
      - "--acme.httpChallenge.entryPoint=http"
      - "--acme.OnHostRule=true"
      - "--acme.onDemand=false"
      - "--acme.email=email@email.com"
      - "--docker"
      - "--docker.swarmmode"
      - "--docker.domain=domain.com"
      - "--docker.watch"
      - "--consul"
      - "--consul.endpoint=consul:8500"
      - "--consul.prefix=traefik"
    networks:
      - traefik
    deploy:
      restart_policy:
        condition: on-failure
    depends_on:
      - consul
  traefik:
    image: traefik:1.5
    depends_on:
      - traefik_init
      - consul
    command:
      - "--consul"
      - "--consul.endpoint=consul:8500"
      - "--consul.prefix=traefik"
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock
    networks:
      - webgateway
      - traefik
    ports:
      - 80:80
      - 443:443
      - 8080:8080
    deploy:
      mode: global
      placement:
        constraints:
          - node.role == manager
      update_config:
        parallelism: 1
        delay: 10s
      restart_policy:
        condition: on-failure
  consul:
    image: consul
    command: agent -server -bootstrap-expect=1
    volumes:
      - consul-data:/consul/data
    environment:
      - CONSUL_LOCAL_CONFIG={"datacenter":"us_west2","server":true}
      - CONSUL_BIND_INTERFACE=eth0
      - CONSUL_CLIENT_INTERFACE=eth0
    deploy:
      replicas: 1
      placement:
        constraints:
          - node.role == manager
      restart_policy:
        condition: on-failure
    networks:
      - traefik

networks:
  webgateway:
    driver: overlay
    external: true
  traefik:
    driver: overlay

volumes:
  consul-data:

Modify the values to reflect what you need – LetsEncrypt wants an email address, and Traefik wants to know about a default domain.  This is arbitrary, since you can point multiple domains and subdomains here and it will all be handled.  Make sure your networks are named corretly (the Traefik network is fine the way it is, and is just a way to talk to Consul).  Make sure you set the Consul datacenter setting correctly.

Once ready, run:

docker stack deploy -c <name of stack file>.yml traefik

And everything should start smoothly.  Traefik now monitors the docker socket and will publish any services that start up and have the appropriate labels on them (read more about labels and docker on the Traefik site).  Essentially, it will auto discover publishable services, and on the first request to them, generate an SSL cert.  You also don’t need to touch to load balancer for any of this, since it listens to the swarm and automatically adds listeners for any published ports!

Traefik has a simple dashboard you can view on port 8080.  If you have a domain pointed to your load balancer, you can hit it on port 8080, or you can hit your load balancer directly on 8080.  I recommend adding a security policy to protect unwanted viewing.

You are now up and running.  As an example, I pointed a domain and all it’s subdomains to the swarm load balancer.  I then deployed a stack that had the labels for Traefik indicating it wanted to discovered on a certain subdomain of that main domain.  once it started, the Traefik system picked up on it and showed it as being active on the dashboard.  I then went to that subdomain, and after a few seconds and requests to generate the SSL, had a fully secure connection.  I can then scale that service in the stack up and down and it round robins perfectly.

Here’s an examle whoami yml.  You would set your zone file to point domain.com to the load balancer and *.domain.com as a CNAME to your main domain, then use this:

version: "3"
services:
    whoami:
        networks:
            - webgateway
        image: emilevauge/whoami
        deploy:
            labels:
                traefik.docker.network: webgateway
                traefik.frontend.rule: Host:whoami.domain.com
                traefik.port: 80
networks:
    webgateway:
        external: true

Run this using the same setup as the Traefik command above (sending your commands to your remote docker swarm):

docker stack deploy -c <whoami file>.yml whoami

And you will see it appear on the Traefik dashboard.  Hit the URL – it may take a few requests, or seconds, for the ssl to work correctly, and then it is all set!

You can now scale the service running:

docker service scale whoami_whoami=3

And it will scale it up!  Hit the domain again and you can see the node ID changing, but the cert working.

Now you are all set to start deploying multiple sites and services to your swarm.