Self-Hosting Handbook: A Docker-Compose Tutorial

Welcome to the second page of a handbook on self-hosting. Begin here.

---

Topics covered on this page

1. How a reverse proxy works and why you need one
2. Creating and understanding docker-compose files
3. Fleshing out your docker-compose file
4. Hitting the "up -d" button on your self-hosted stack

How a reverse proxy works and why you need one

A reverse proxy is a web server that accepts incoming traffic, analyzes it, and directs it to the appropriate backend service/server. Reverse proxies are often used as load balancers or caching servers, but for the sake of this handbook, we're mostly interested in how a reverse proxy can accept traffic from multiple domains, all directed toward the same VPS, and deliver the appropriate content in turn.

Let's look at this example diagram:

ssdnodes.com ----|                     |-- LAMP stack + PHP files
                 |    reverse proxy    |
x.ssdnodes.com --|---       @       ---|-- WordPress blog
                 |    123.456.78.90    |
y.ssdnodes.com --|                     |-- Node.js web app

In this case, we have three domains—ssdnodes.com, x.ssdnodes.com, and y.ssdnodes.com—that are all pointed toward a VPS at the IP address 123.456.78.90. We also have three backend services—a LAMP stack, a WordPress blog, and a Node.js web app—that we want to serve.

When someone types one of those three domains into their browser's navigation bar, they send a request to the VPS via DNS (remember DNS from the previous page?). The reverse proxy receives this request, analyzes it, and recognizes that the request is looking for ssdnodes.com. The reverse proxy then sends a request to the LAMP stack, which returns an HTML page. Finally, the reverse proxy "forwards" that HTML page (proxies it, actually, hence the name) to the user's browser.

When they type x.ssdnodes.com into their browser, the reverse proxy returns the WordPress blog, and when the type x.ssdnodes.com into their browser, they get the Node.js app in return.

In this way, a reverse proxy allows you to host multiple websites or web apps from a single VPS.

I consider a reverse proxy necessary to self-hosting because it's inconvenient enough to remember the IP address of your VPS, much less the various ports on which each of your services will be running. Also, the reverse proxy will request and serve the Let's Encrypt SSL certificate that will allow you to have HTTPS-enabled websites and apps.

Why is SSL important? Expand

Creating and understanding docker-compose files

Now that you have some context for what we're trying to build with this self-hosting stack, it's time to take a look at the primary tool we're going to use moving forward: docker-compose.

You installed this program on the previous page. docker-compose is a "wrapper" around the docker command-line program itself, and uses configuration files called docker-compose.yml to launch multiple Docker containers instead of typing out long commands with docker.

Log into your VPS if you're not already, create a new proxy directory in your home directory, and cd into it.

$ mkdir -p ~/proxy
$ cd ~/proxy

Create a new file using the editor of your choice. I use nano for these kinds of tasks, but you may prefer to use vim or emacs instead. Just replace nano with your editor of choice from here on out.

$ nano docker-compose.yml

Let's start with building out the "skeleton" of a docker-compose.yml file, which will then allow us to talk about how each component works, and the syntax behind it. Inside the new file, type in the following.

version: '2'

services:

volumes:

networks:

Explanation: version

As docker-compose expands and becomes more feature-rich, it needs to evolve and adopt new ways of explaining how users can deploy a Docker-based infrastructure. The maintainers infrequently introduce a new version of the syntax to accommodate these adaptations. We're sticking with the '2' syntax, even though there's already a third version version: '3' available already.

Version 3 introduces a new way of describing volumes that I haven't gotten working precisely the way I'd like yet. I'll be sure to update this guide when I have a version 3-compatible version of this docker-compose.yml file ready.

Explanation: services

The services section defines which Docker containers you'll launch when you finally do run docker-compose. It's the most important section we'll cover in on the rest of this page in the self-hosting handbook!

We will define each service in a separate indented "block." When we run docker-compose, it will iterate over these services blocks and download/create/launch each Docker container in sequence.

Explanation: volumes

Volumes are essentially folders that get shared between your user's filesystem and the Docker container's filesystem. Think of it as a symbolic link. Or, if you're not too familiar with those, think about it as a Dropbox folder on your current machine. You put something in that folder, and it appears on the Dropbox web interface, or on another device of yours—the folders are synchronized.

Because volumes are synchronized, you can define various configuration files and make changes to them without diving into the Docker container's filesystem, which can be quite tricky.

You can also create a single volume that's shared among many Docker containers, such as using one MySQL database for five different WordPress installations. But that's, well, not recommended. Not by a long shot.

Explanation: networks

The networks section in a docker-compose file allows you to define internal networks through which different Docker containers can communicate. We're going to keep it simple throughout this handbook, with only a single network, but more complex infrastructures might require multiple internal/external networks.

Fleshing out your docker-compose file

Time to build out the docker-compose.yml file!

Under the services block, add your first service, which we're going to call proxy. Remember that indentation is important with docker-compose files—the services label should be aligned fully to the left, while each defined service is indented once, and its contents are indented twice or three times.

services:

  proxy:
    image: jwilder/nginx-proxy
    container_name: proxy
    restart: unless-stopped
    labels:
      com.github.jrcs.letsencrypt_nginx_proxy_companion.nginx_proxy: "true"
    volumes:
      - /var/run/docker.sock:/tmp/docker.sock:ro
      - certs:/etc/nginx/certs:rw
      - vhost.d:/etc/nginx/vhost.d
      - html:/usr/share/nginx/html
      - ./uploadsize.conf:/etc/nginx/conf.d/uploadsize.conf:ro
    ports:
      - "80:80"
      - "443:443"
    networks:
      - "default"
      - "proxy-tier"

Let's take a minute to look at what this services block accomplishes.

  proxy

The first line of the block defines a new Docker container. You can name them however you'd like, but let's keep it simple.

    image: jwilder/nginx-proxy

This line defines the Docker image that we're going to launch—in this case, the wonderful nginx-proxy image.

    container_name: proxy

We're going to name the container proxy in this case, and refer to it as that moving forward, but you can name it however you'd like.

    labels:
      com.github.jrcs.letsencrypt_nginx_proxy_companion.nginx_proxy: "true"

This nested labels line creates a label so that the Let's Encrypt container, which we'll launch in a moment, knows which proxy to connect to.

    volumes:
      - /var/run/docker.sock:/tmp/docker.sock:ro
      - certs:/etc/nginx/certs:rw
      - vhost.d:/etc/nginx/vhost.d
      - html:/usr/share/nginx/html

Here we create some volumes for our proxy Docker container. First, we connect the Docker .sock, followed by three additional volumes that create those "synchronized" folders between the proxy folder we created earlier and various folders inside of the container.

    ports:
      - "80:80"
      - "443:443"

We're now defining which ports the Docker container will listen on. By listening on ports 80 and 443, the proxy container will accept all web traffic arriving on your VPS, which is exactly what we want—the reverse proxy needs to receive all the traffic so that it can route it correctly.

    networks:
      - "default"
      - "proxy-tier"

Finally, we're defining which networks the proxy container will communicate through. The default network will handle traffic entering and exiting the VPS, whereas the proxy-tier network allows all the containers in our self-hosting stack to communicate with one another.

Let's Encrypt container

Time to add the Let's Encrypt container to the file.

  proxy-letsencrypt:
    image: jrcs/letsencrypt-nginx-proxy-companion
    container_name: letsencrypt
    restart: unless-stopped
    environment:
      - NGINX_PROXY_CONTAINER=proxy
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock:ro
    volumes_from:
      - "proxy"
    depends_on:
      - "proxy"
    networks:
      - "default"
      - "proxy-tier"

There are a few new configurations here, namely environment, volumns_from and depends_on. All these help the Let's Encrypt container find and connect to the proxy container we just specified. You shouldn't need to edit these, so we won't spend time discussing the nuances of how they work.

Portainer container

I like to add a Portainer container to any self-hosting stack as a means of monitoring my containers and doing some basic administrative work that's much more difficult using the many options and flags of the docker command line tool.

As a bonus, this portion of the docker-compose.yml file will serve as an example of how we'll eventually set up other self-hosted services.

Time to add one last block to our services section.

  portainer:
    image: portainer/portainer
    container_name: portainer
    restart: always
    environment:
      - VIRTUAL_HOST=docker.DOMAIN.TLD
      - LETSENCRYPT_HOST=docker.DOMAIN.TLD
      - LETSENCRYPT_EMAIL=EMAIL
    volumes:
      - ./portainer/:/data
      - /var/run/docker.sock:/var/run/docker.sock
    ports:
      - "9000:9000"

Take a look at the environment block here, as it's critical to how this entire stack functions, from the reverse proxy to how Docker containers communicate with one another. You will put a similar block in every new container you add to the services section of your docker-compose.yml file.

There are three essential environment variables to set here: the VIRTUAL_HOST, the LETSENCRYPT_HOST, and the LETSENCRYPT_EMAIL.

The LETSENCRYPT_EMAIL variable is simplest to explain: Replace the EMAIL variable with your email.

The VIRTUAL_HOST and LETSENCRYPT_HOST both reference the domain at which you'll access the Portainer service using your browser. Both these variables must be set on every Docker container you want to access via the reverse proxy. On the previous page, you set up a DNS entry that pointed toward docker.DOMAIN.TLD. Here's where that DNS entry comes in. Flesh out both those entries with your domain name.

Volumes and networks

Finally, add the following to your docker-compose.yml file:

volumes:
  certs:
  vhost.d:
  html:

networks:
  proxy-tier:

We won't spend too much time on these, as they don't need to be edited and will never change, but they set up a number of volumes and a single proxy-tier network that allows containers to communicate amongst themselves.

The full file, for reference

Knowing that copy-pasting or re-typing configuration files is a rather error-prone process, here is the full file for your reference:

version: '2'

services:

  proxy:
    image: jwilder/nginx-proxy
    container_name: proxy
    restart: unless-stopped
    labels:
      com.github.jrcs.letsencrypt_nginx_proxy_companion.nginx_proxy: "true"
    volumes:
      - /var/run/docker.sock:/tmp/docker.sock:ro
      - certs:/etc/nginx/certs:rw
      - vhost.d:/etc/nginx/vhost.d
      - html:/usr/share/nginx/html
      - ./uploadsize.conf:/etc/nginx/conf.d/uploadsize.conf:ro
    ports:
      - "80:80"
      - "443:443"
    networks:
      - "default"
      - "proxy-tier"

  proxy-letsencrypt:
    image: jrcs/letsencrypt-nginx-proxy-companion
    container_name: letsencrypt
    restart: unless-stopped
    environment:
      - NGINX_PROXY_CONTAINER=proxy
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock:ro
    volumes_from:
      - "proxy"
    depends_on:
      - "proxy"
    networks:
      - "default"
      - "proxy-tier"

  portainer:
    image: portainer/portainer
    container_name: portainer
    restart: always
    environment:
      - VIRTUAL_HOST=docker.DOMAIN.TLD
      - LETSENCRYPT_HOST=docker.DOMAIN.TLD
      - LETSENCRYPT_EMAIL=EMAIL
    volumes:
      - ./portainer/:/data
      - /var/run/docker.sock:/var/run/docker.sock
    ports:
      - "9000:9000"

volumes:
  certs:
  vhost.d:
  html:

networks:
  proxy-tier:

If you copy-paste this entire file, you must still replace the EMAIL, DOMAIN, and TLD variables before moving forward.

Allowing for big file uploads

Before you can launch the stack, create one more file in your proxy directory called uploadsize.conf. Inside of it, add a single line with the following: client_max_body_size 1g;.

Our reverse proxy will link this file to the nginx configuration and allow you to upload files up to 1GB in size to your self-hosting stack. Very useful when using a file synchronization service like Nextcloud.

Hitting the "up -d" button on your self-hosting stack

Now that the docker-compose.yml file is finished, it's time to run docker-compose for the first time. Type the following into your VPS' terminal:

$ docker-compose up -d

Hit Enter, and you'll see quite a bit of output as Docker downloads the images. If all goes correctly, you should see the following as the process ends:

Creating network "proxy_default" with the default driver
Creating network "proxy_proxy-tier" with the default driver
Creating portainer ... done
Creating proxy     ... done
Creating letsencrypt ... done

Let's do a quick check of our self-hosted stack to make sure that everything is running correctly. Run docker ps and you should see the following:

CONTAINER ID        IMAGE                                    COMMAND                  CREATED   STATUS              PORTS                                      NAMES
bf830886f0a2        jrcs/letsencrypt-nginx-proxy-companion   "/bin/bash /app/entr…"   3 minutes ago   Up 3 minutes                                                   letsencrypt
5d0845395978        jwilder/nginx-proxy                      "/app/docker-entrypo…"   3 minutes ago   Up 3 minutes        0.0.0.0:80->80/tcp, 0.0.0.0:443->443/tcp   proxy
10bd13114c5d        portainer/portainer                      "/portainer"             3 minutes ago   Up 3 minutes        0.0.0.0:9000->9000/tcp                     portainer

The output shows that nginx-proxy, the Let's Encrypt helper, and Portainer are all running.

Open up a browser and navigate to docker.DOMAIN.TLD. You should see the Portainer setup process—create an administrative account, log in, and you should be able to see all your containers running in the nice web UI.

If you made it this far, congratulations! You're already 75% of your way to having a fully-functional self-hosting stack on your VPS.

From here on out, I'll explain how Let's Encrypt is working, why that's important, and how you can add more containers. After that, I'll start creating a small "database" of example services blocks for you to pick and choose from.

---

Bookmark this guide and follow us on Twitter or Mastodon to get updates. Or, you can subscribe to the weekly Serverwise newsletter, where I’ll let you know as soon as this guide expands.

[cta]