Docker Images Not Starting

After updating my Linux host yesterday, the docker images failed to start with the following error:

Error starting userland proxy: listen tcp6 [::]:9091: socket: address family not supported by protocol

For some reason IPv6 (the hint is tcp6) is screwing things up. The problem is that I disabled IPv6 from the start on this host. Mainly because of some concerns in regards of routing and internet accessibility (I have a formal IPv6 subnet at home).

After about an hour of troubleshooting I changed the docker-compose.yaml file to include the actual IP address of the host instead of just the ports the container uses.

The old:

        ports:
          - '9091:9091'
          - '8888:8080'

New:

        ports:
          - '192.168.0.1:9091:9091'
          - '192.168.0.1:8888:8080'

Restarting the docker images went just fine after this. So I guess the update I ran yesterday included a docker update that basically thinks that you have IPv6 enabled by default. The problem is that I couldn’t find documentation on how to disable this globally.

My First Docker Deployment

About two weeks ago I had to get a crash-course in Docker technology at work. I had no idea what I was doing (following a YouTube video and an accompanying PDF. Eventually I got it to work but no idea what I was doing. So I wanted to change that.

NOTE: I’m not gonna build/create new docker containers yet. Just learning on how they are used, configured, and interact. This is also no tutorial on how to install Docker itself. There are more than enough websites for that.

The problem with learning new things is that they have to be practically and/or useful (for me). After some thought I ended up with a combination of Transmission and OpenVPN.

This Docker image gives you a Torrent client with a webgui, and all (torrent) traffic is directed through the OpenVPN connection. Making it safe to download Linux distro’s. As a bonus, it has a generic web proxy function with you can use to handle your web traffic. The latter is especially useful in combination with e.g. the browser extension/plugin FoxyProxy.

Deploying the docker container is pretty straight-forward (you do need a supported VPN provider). It basically works out of the box, but molding it to my wishes involved a bit more digging around. There are some things I wanted to add, or change;

• Customer paths for the download locations (default = /data).

• Use a watchfolder for transmission where the .torrent files can be picked-up.

• Use an additional reverse proxy for the Transmission webGui so that all my internal services are accessible from 1 IP address without having to remember al their TCP ports.
  More info on that can be found here.

• Since the Docker container runs under a/the root account, I needed to change that behavior since I don’t want to do everything with root permissions (involving experimentation with umask and UID/GID’s).

This resulted in the following docker-compose file (docker-compose.yaml):

UPDATE: I’ve added the Portainer image to the compose file. This gives you a web gui to manage the containers. The gui is accessible on port 9000 on the same docker host.

version: '2'
services:
    transmission-openvpn:
        restart: unless-stopped
        volumes:
            - '/mnt/data:/data'
            - '/mnt/stack/Watchfolder:/home/Watchfolder'
            - '/etc/localtime:/etc/localtime:ro'
        environment:
            - TZ=Europe/Amsterdam
            - OPENVPN_OPTS=--inactive 3600 --ping 10 --ping-exit 60
            - CREATE_TUN_DEVICE=true
            - OPENVPN_PROVIDER=NORDVPN 
            - OPENVPN_USERNAME=<VPN-USERNAME>
            - OPENVPN_PASSWORD=<VPN-PASSWORD>
            - NORDVPN_COUNTRY=CH
            - HEALTH_CHECK_HOST=google.com
            - TRANSMISSION_INCOMPLETE_DIR_ENABLED=true
            - TRANSMISSION_INCOMPLETE_DIR=/data/downloads/incomplete
            - TRANSMISSION_DOWNLOAD_DIR_ENABLED=true
            - TRANSMISSION_DOWNLOAD_DIR=/data/downloads/complete
            - TRANSMISSION_WATCH_DIR_ENABLED=true
            - TRANSMISSION_WATCH_DIR=/home/Watchfolder
            - TRANSMISSION_TRASH_ORIGINAL_TORRENT_FILES=true
#            - TRANSMISSION_UMASK=222
            - TRANSMISSION_SPEED_LIMIT_DOWN=5000
            - TRANSMISSION_SPEED_LIMIT_DOWN_ENABLED=true
            - TRANSMISSION_SPEED_LIMIT_UP=1000
            - TRANSMISSION_SPEED_LIMIT_UP_ENABLED=true
            - WEBPROXY_ENABLED=true
            - WEBPROXY_PORT=8080
            - LOCAL_NETWORK=192.168.0.0/16
            - PUID=1000
            - PGID=1000
        cap_add:
            - NET_ADMIN
        logging:
            driver: json-file
            options:
                max-size: 10m
        ports:
                - '9091:9091'
                - '8888:8080'
        image: haugene/transmission-openvpn
        container_name: openvpn
    portainer:
        image: portainer/portainer-ce
        container_name: portainer
        restart: always
        ports:
          - "9000:9000"
        command: -H unix:///var/run/docker.sock
        volumes:
          - /var/run/docker.sock:/var/run/docker.sock
          - portainer_data:/data

volumes:
  portainer_data:

The thing that took the most amount of time to figure out was the PUID/PGID part of the config. This basically are the user id and group id which are used to run the container and also when creating directories and files on the physical filesystem of the host. In my case, the PUID, and PGID are the id’s corresponding to my username on the Linux host.

The important part is that all the path references in the environment part of the yaml file are local to the container. These are mapped/related to the physical locations in the volumes part of the config file.

Deploying/creating the Docker container is done through the following command (I use docker-compose instead of docker run):

docker-compose up -d

Configuring a webproxy in my browser pointing to the Linux host IP with port 8888 allows me to surf the web through the OpenVPN provider. Pointing my browser to the Linux host IP address with port 9091 gives the Transmission webGui (http://IP-ADDRESS>:9091). But as I mentioned earlier, I want to access this through my internal reverse proxy (NGINX).

To do this I have to create an additional location within the NGINX config and enable that. This resulted in the following NGINX location config file:

location /transmission/ {
      proxy_pass http://192.168.##.1:9091;

      proxy_set_header X-Real-IP $remote_addr;
      proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
      proxy_set_header Host $http_host;
      proxy_set_header X-NginX-Proxy true;
      proxy_http_version 1.1;
      proxy_set_header Connection "";
      proxy_pass_header X-Transmission-Session-Id;
      add_header   Front-End-Https   on;

      satisfy any;
      allow 192.168.0.0/16;
      allow 172.16.16.0/24;
      allow 10.200.200.200/32;
      deny all;
      auth_basic "Restricted Content";
      auth_basic_user_file /etc/nginx/auth.d/auth.pwd;
}

Note that the bottom part are some directoves to limit the IP’s that can access the page. These are related to your internal IP networks.

Now I can access the tranmission webGui over https through my NGINX reverse proxy via https://internalhost/transmission

A small note on the use of FoxyProxy; This extension allows you to selectively use the proxy based on (parts of ) URL. You can configure patterns using wildcards and regular expressions to direct traffic directly or through a proxy.

So if e.g. your OpenVPN terminates in the the US, you can create a pattern that certain entertainment sites are being accessed through your proxy, while other traffic uses your regular ISP. This is especially useful if you have a capped monthly VPN account.

Increase Ubuntu VM Disk Space

I ran into a very familiar problem (for me anyway) on an Ubuntu 18 virtual machine today. It started with, what I thought was, a Cisco ISE issue. Cisco ISE reports that I wanted to write to a repository ended up as empty files (0 bytes). Initial thought was that I ran into an Cisco ISE bug related to repositories, but it turned out that my Ubuntu server had run out of disk space;

Err:2 http://archive.ubuntu.com/ubuntu bionic-updates InRelease
  Error writing to output file - write (28: No space left on device) [IP: 91.189.88.24 80]

A quick df -h revealed that my Ubuntu server had only 4GB (out of the 100GB I assigned in VMWare).

willem@ubuntu:~$ df -h
Filesystem                         Size  Used Avail Use% Mounted on
udev                               967M     0  967M   0% /dev
tmpfs                              200M  1.1M  199M   1% /run
/dev/mapper/ubuntu--vg-ubuntu--lv  3.9G  3.9G    0M 100% /
tmpfs                              997M     0  997M   0% /dev/shm
tmpfs                              5.0M     0  5.0M   0% /run/lock
tmpfs                              997M     0  997M   0% /sys/fs/cgroup
/dev/loop0                          90M   90M     0 100% /snap/core/7713
/dev/loop1                          90M   90M     0 100% /snap/core/7917
/dev/sda2                          976M  222M  688M  25% /boot
tmpfs                              200M     0  200M   0% /run/user/1000

So, now I had to add the unassigned 96GB to the Ubuntu filesystem. Thankfully, askubuntu to the rescue.

root@ubuntu:~# lvm
lvm> lvextend -l +100%FREE /dev/mapper/ubuntu--vg-ubuntu--lv
  Size of logical volume ubuntu-vg/ubuntu-lv changed from 4.00 GiB (1024 extents) to <99.00 GiB (25343 extents).
  Logical volume ubuntu-vg/ubuntu-lv successfully resized.
lvm> exit
  Exiting.
root@ubuntu:~# resize2fs /dev/mapper/ubuntu--vg-ubuntu--lv
resize2fs 1.44.1 (24-Mar-2018)
Filesystem at /dev/mapper/ubuntu--vg-ubuntu--lv is mounted on /; on-line resizing required
old_desc_blocks = 1, new_desc_blocks = 13
The filesystem on /dev/mapper/ubuntu--vg-ubuntu--lv is now 25951232 (4k) blocks long.

root@ubuntu:~# df -h
Filesystem                         Size  Used Avail Use% Mounted on
udev                               967M     0  967M   0% /dev
tmpfs                              200M  1.1M  199M   1% /run
/dev/mapper/ubuntu--vg-ubuntu--lv   98G  3.4G   91G   4% /
tmpfs                              997M     0  997M   0% /dev/shm
tmpfs                              5.0M     0  5.0M   0% /run/lock
tmpfs                              997M     0  997M   0% /sys/fs/cgroup
/dev/loop0                          90M   90M     0 100% /snap/core/7713
/dev/loop1                          90M   90M     0 100% /snap/core/7917
/dev/sda2                          976M  222M  688M  25% /boot
tmpfs                              200M     0  200M   0% /run/user/1000
root@ubuntu:~# 

Add Routes To Ubuntu Server

Routes (non-default ones) can be added to the platform by using the interfaces configurations file.

Just add the following to the interface configuration:

up route add -net <destination_network> netmask <netmask> gw <gateway_address>

Example:

$ cat /etc/network/interfaces
# This file describes the network interfaces available on your system
# and how to activate them. For more information, see interfaces(5).

source /etc/network/interfaces.d/*

# The loopback network interface
auto lo
iface lo inet loopback

# The primary network interface
auto ens160
iface ens160 inet static
        address 192.168.168.1
        netmask 255.255.255.0
        network 192.168.168.0
        broadcast 192.168.168.255
        gateway 192.168.168.254
        # dns-* options are implemented by the resolvconf package, if installed
        dns-nameservers 192.168.0.1
        dns-search mydomain.com
        up route add -net 172.16.16.0 netmask 255.255.255.0 gw 192.168.168.30

Capture Network Traffic With Wireshark Under Ubuntu

When you install Wireshark on Ubuntu Linux you need to be root to be able to capture traffic. The standard user doesn't have enough privileges to do this. 

A workaround for this is to add the user to a wireshark group and give the group special permissions. Afterwards, you're able to cpature traffic in Ubuntu with Wireshark, without needing root access. 

The complete list of commands: 

sudo groupadd wireshark

sudo usermod -a -G wireshark <YOUR-USER-NAME>

sudo chgrp wireshark /usr/bin/dumpcap

sudo chmod 750 /usr/bin/dumpcap

sudo setcap cap_net_raw,cap_net_admin=eip /usr/bin/dumpcap

sudo getcap /usr/bin/dumpcap

Just reboot, or log out and back in, and you're finished. 

HAPPY CAPTURING

Courier IMAPd and Mail.app warnings

After installing an ISPConfig deployment, everything seemed to work properly, but every now-and-then I got this weird error that there was something wrong with the mail server configuration. The Apple Mail.app showed a exclamation mark with the following message:

The server returned the error: The attempt to read data from the server server.domain.ext failed.

Some research showed that the Apple mail clients tend to open several connections for IMAP, and the default setting of the Courier IMAPd server is to allow (only) 4 connections from the same IP address.

Modifying the Courier config file (/usr/lib/courier-imap/etc/imapd) and allowing e.g. 20 connections from 1 IP address solved this problem.

<ORIGINAL CONFIG>
##NAME: MAXPERIP:0
#
#  Maximum number of connections to accept from the same IP address

MAXPERIP=4

<MODIFIED CONFIG>
##NAME: MAXPERIP:0
#
#  Maximum number of connections to accept from the same IP address

MAXPERIP=20

 If your company / household holds several imap mail clients you may need to increase the counter even more (65536 is the maximum amount of connections for ANY IP address).

If you have SSL enabled on the Courier IMAPd server you also need to add the MAXPERIP variable to the imap-ssl config file (/usr/lib/courier-imap/etc/imapd-ssl).

Finally, you need to restart the Courier IMAPd services (/etc/init.d/courier-imap restart)

Getting ISPConfig to Work on Centos

This is not a manual describing the installation (pre-requisites) of ISPConfig software on a Centos platform. An excellent manual can be found online. It's just that I ran into a problem when I tried to connect an e-mail client to the (IMAP) mailserver (controled by ISPConfig). All the appropriate ports / listeners were up and running, so it had to be a configuration issue.

Googling around didn't solve my problem. My collegue, Xander (@xmoments / xmoments.nl), cam eto the rescue with the solution;

yum install cyrus-sasl-plain-2.1.23-13.el6.x86_64

Software that handles cleartext passwords between mail processes. After the installation, the mail went flying across the Interwebs.