Difference between revisions of "Docker"

Latest revision as of 21:40, 3 March 2023

Docker is an open-source project that automates the deployment of applications inside software containers. Quote of features from docker web page:

Docker containers wrap up a piece of software in a complete filesystem that contains everything it needs to run: code, runtime, system tools, system libraries – anything you can install on a server. This guarantees that it will always run the same, regardless of the environment it is running in.^[1]

Introduction

Note: The following is based on content found on the official Docker website, Wikipedia, and various other locations.

A container image is a lightweight, stand-alone, executable package of a piece of software that includes everything needed to run it: code, runtime, system tools, system libraries, settings. Available for both Linux and Windows based apps, containerized software will always run the same, regardless of the environment. Containers isolate software from its surroundings, for example differences between development and staging environments and help reduce conflicts between teams running different software on the same infrastructure.

Lightweight 

Docker containers running on a single machine share that machine's operating system kernel; they start instantly and use less compute and RAM. Images are constructed from filesystem layers and share common files. This minimizes disk usage and image downloads are much faster.

Standard 

Docker containers are based on open standards and run on all major Linux distributions, Microsoft Windows, and on any infrastructure including VMs, bare-metal and in the cloud.

Secure 

Docker containers isolate applications from one another and from the underlying infrastructure. Docker provides the strongest default isolation to limit app issues to a single container instead of the entire machine.

As actions are done to a Docker base image, union file-system layers are created and documented, such that each layer fully describes how to recreate an action. This strategy enables Docker's lightweight images, as only layer updates need to be propagated (compared to full VMs, for example).

Building on top of facilities provided by the Linux kernel (primarily cgroups and namespaces), a Docker container, unlike a virtual machine, does not require or include a separate operating system. Instead, it relies on the kernel's functionality and uses resource isolation for CPU and memory, and separate namespaces to isolate the application's view of the operating system. Docker accesses the Linux kernel's virtualization features directly using the libcontainer library (written in the Go programming language).

Comparing Containers and Virtual Machines

Containers and virtual machines have similar resource isolation and allocation benefits, but function differently because containers virtualize the operating system instead of hardware. Containers are more portable and efficient.

Virtual Machines 

Virtual machines (VMs) are an abstraction of physical hardware turning one server into many servers. The hypervisor allows multiple VMs to run on a single machine. Each VM includes a full copy of an operating system, one or more apps, necessary binaries and libraries - taking up tens of GBs. VMs can also be slow to boot.

Containers 

Containers are an abstraction at the app layer that packages code and dependencies together. Multiple containers can run on the same machine and share the OS kernel with other containers, each running as isolated processes in user space. Containers take up less space than VMs (container images are typically tens of MBs in size), and start almost instantly.

Components

The Docker software as a service offering consists of three components:

Software 

The Docker daemon, called "dockerd" is a persistent process that manages Docker containers and handles container objects. The daemon listens for API requests sent by the Docker Engine API. The Docker client, which identifies itself as "docker", allows users to interact with Docker through CLI. It uses the Docker REST API to communicate with one or more Docker daemons.

Objects 

Docker objects refer to different entities used to assemble an application in Docker. The main Docker objects are images, containers, and services.

• A Docker container is a standardized, encapsulated environment that runs applications. A container is managed using the Docker API or CLI.
• A Docker image is a read-only template used to build containers. Images are used to store and ship applications.
• A Docker service allows containers to be scaled across multiple Docker daemons. The result is known as a "swarm", cooperating daemons that communicate through the Docker API.

Registries 

A Docker registry is a repository for Docker images. Docker clients connect to registries to download ("pull") images for use or upload ("push") images that they have built. Registries can be public or private. Two main public registries are Docker Hub and Docker Cloud. Docker Hub is the default registry where Docker looks for images.

Docker commands

I will provide detailed examples on all of the following commands throughout this article.

Basics

The following are the most common Docker commands (i.e., the ones you will most likely use the most day-to-day):

• Show all running containers:

$ docker ps

• Show all containers (including stopped and failed ones):

$ docker ps -a

• Show all images in your local repository:

$ docker images

• Create an image based on the instructions in a Dockerfile:

$ docker build

• Start a container from an image (either from your local repository or from a remote repository {e.g., Docker Hub}):

$ docker run

• Remove/delete all stopped/failed containers (leaves running containers alone):

$ docker rm $(docker ps -a -q)

Container commands

Container lifecycle

• Create a container but do not start it:

$ docker create

• Rename a container:

$ docker rename

• Create and start a container in one operation:

$ docker run

• Delete a container:

$ docker rm

• Update a container's resource limits:

$ docker update

Starting and stopping containers

• Start a container:

$ docker start

• Stop a running container:

$ docker stop

• Stop and start start a container:

$ docker restart

• Pause a running container ("freeze" it in place):

$ docker pause

• Un-pause a paused container:

$ docker unpause

• Attach/connect to a running container:

$ docker attach

• Block until running container stops (and print exit code):

$ docker wait

• Send SIGKILL to a running container:

$ docker kill

Information

• Show all running containers:

$ docker ps

• Get the logs for a given container:

$ docker logs

• Get all of the metadata about a container (e.g., IP address, etc.):

$ docker inspect

• Get real-time events from Docker Engine (e.g., start/stop containers, attach, create, etc.):

$ docker events

• Get the public-facing ports of a given container:

$ docker port

• Show running processes in a given container:

$ docker top

• Show a given container's resource usage statistics:

$ docker stats

• Show changed files in the container's filesystem (i.e., those changed from the original base image):

$ docker diff

Miscellaneous

• Get the environment variables for a given container:

$ docker run ubuntu env

• IP address of host machine:

$ ip -4 -o addr show eth0
2: eth0    inet 10.0.0.166/23

• IP address of a container:

$ docker run ubuntu ip -4 -o addr show eth0
2: eth0    inet 172.17.0.2/16

Image commands

Lifecycle

• Show all images in your local repository:

$ docker images

• Create an image from a tarball:

$ docker import

• Create an image from a Dockerfile

$ docker build

• Create an image from a container (note: it will pause the container, if it is running, during the commit process):

$ docker commit

• Remove/delete an image:

$ docker rmi

• Load an image from a tarball as STDIN (including images and tags):

$ docker load

• Save an image to a tarball (streamed to STDOUT with all parents lays, tags, and versions):

$ docker save

Info

• Show the history of an image:

$ docker history

• Tag an image:

$ docker tag

Dockerfile directives

USER

$ cat << EOF > Dockerfile
# Non-privileged user entry
FROM centos:latest
MAINTAINER xtof@example.com

RUN useradd -ms /bin/bash xtof
USER xtof
EOF

Note: The use of MAINTAINER has been deprecated in newer versions of Docker. You should use LABEL instead, as it is much more flexible and its key/values show up in docker inspect. From here forward, I will only use LABEL.

$ docker build -t centos7/nonroot:v1 .
$ docker exec -it <container_name> /bin/bash

We are user "xtof" and are unable to become root. The workaround (i.e., how to become root) is like so:

$ docker exec -u 0 -it <container_name> /bin/bash

NOTE: For the remainder of this section, I will omit the $ cat << EOF > Dockerfile part in the examples for brevity.

RUN

Notes on the order of execution

FROM centos:latest
LABEL maintainer="xtof@example.com"

RUN useradd -ms /bin/bash xtof
USER xtof

RUN echo "export PATH=/path/to/my/app:$PATH" >> /etc/bashrc

$ docker build -t centos7/config:v1 .
...
/bin/sh: /etc/bashrc: Permission denied

The order of execution matters! Prior to the directive USER xtof, the user was root. After that directive, the user is now xtof, who does not have super-user privileges. Move the RUN echo ... directive to before the USER xtof directive for a successful build.

ENV

Note: The following is a _terrible_ way of building a container. I am purposely doing it this way so I can show you a much better way later (see below).

• Build a CentOS 7 Docker image with Java 8 installed:

# SEE: https://gist.github.com/P7h/9741922 for various Java versions
FROM centos:latest
LABEL maintainer="xtof@example.com"

RUN yum update -y
RUN yum install -y net-tools wget

RUN echo "SETTING UP JAVA"
# The tarball method:
#RUN cd ~ && wget --no-cookies --no-check-certificate \
#    --header "Cookie: gpw_e24=http%3A%2F%2Fwww.oracle.com%2F; oraclelicense=accept-securebackup-cookie" \
#    "http://download.oracle.com/otn-pub/java/jdk/8u91-b14/jdk-8u91-linux-x64.tar.gz"
#RUN tar xzvf jdk-8u91-linux-x64.tar.gz
#RUN mv jdk1.8.0_91 /opt
#ENV JAVA_HOME /opt/jdk1.8.0_91/

# The rpm method:
RUN cd ~ && wget --no-cookies --no-check-certificate \
    --header "Cookie: gpw_e24=http%3A%2F%2Fwww.oracle.com%2F; oraclelicense=accept-securebackup-cookie" \
    "http://download.oracle.com/otn-pub/java/jdk/8u161-b12/2f38c3b165be4555a1fa6e98c45e0808/jdk-8u161-linux-x64.rpm"
RUN yum localinstall -y /root/jdk-8u161-linux-x64.rpm

RUN useradd -ms /bin/bash xtof
USER xtof

# User specific environment variable
RUN cd ~ && echo "export JAVA_HOME=/usr/java/jdk1.8.0_161/jre" >> ~/.bashrc
# Global (system-wide) environment variable
ENV JAVA_BIN /usr/java/jdk1.8.0_161/jre/bin

$ docker build -t centos7/java8:v1 .

CMD vs. RUN

FROM centos:latest
LABEL maintainer="xtof@example.com"

RUN useradd -ms /bin/bash xtof
CMD ["echo", "Hello from within my container"]

The CMD directive only executes when the container is started, whereas the RUN directive is executed during the build of the image.

$ docker build -t centos7/echo:v1 .
$ docker run centos7/echo:v1
Hello from within my container

The container starts, echos out that message, then exits.

ENTRYPOINT

FROM centos:latest
LABEL maintainer="xtof@example.com"

RUN useradd -ms /bin/bash xtof
ENTRYPOINT "This command will display this message on EVERY container that is run from it"

$ docker build -t centos7/entry:v1 .
$ docker run centos7/entry:v1
This command will display this message on EVERY container that is run from it
$ docker run centos7/entry:v1 /bin/echo "Can you see me?"
This command will display this message on EVERY container that is run from it
$ docker run centos7/echo:v1 /bin/echo "Can you see me?"
Can you see me?

Note the difference.

EXPOSE

FROM centos:latest
LABEL maintainer="xtof@example.com"

RUN yum update -y
RUN yum install -y httpd net-tools

RUN echo "This is a custom index file built during the image creation" > /var/www/html/index.html

ENTRYPOINT apachectl -DFOREGROUND  # BAD WAY TO DO THIS!

$ docker build -t centos7/apache:v1 .
$ docker run -d --name webserver centos7/apache:v1
$ docker exec webserver /bin/cat /var/www/html/index.html
This is a custom index file built during the image creation
$ docker inspect webserver -f '{{.NetworkSettings.IPAddress}}'  # => 172.17.0.6
#~OR~
$ docker inspect webserver | jq -crM '.[] | .NetworkSettings.IPAddress'  # => 172.17.0.6
$ curl 172.17.0.6
This is a custom index file built during the image creation
$ curl -sI 172.17.0.6 | awk '/^HTTP|^Server/{print}'
HTTP/1.1 200 OK
Server: Apache/2.4.6 (CentOS)
$ time docker stop webserver
real	0m10.275s  # <- notice how long it took to stop the container
user	0m0.008s
sys	0m0.000s
$ docker rm webserver

It took ~10 seconds to stop the above container. This is because of the way we are (incorrectly) using ENTRYPOINT. The SIGTERM signal when running `docker stop webserver` actually timed out instead of exiting gracefully. A much better method is shown below, which will exit gracefully and in less than 300 ms.

• Expose ports from the CLI

$ docker run -d --name webserver -p 8080:80 centos7/apache:v1
$ curl localhost:8080
This is a custom index file built during the image creation
$ docker stop webserver && docker rm webserver

• Explicitly expose a port in the Docker image:

FROM centos:latest
LABEL maintainer="xtof@example.com"

RUN yum update -y && \
    yum install -y httpd net-tools && \
    yum autoremove -y && \
    echo "This is a custom index file built during the image creation" > /var/www/html/index.html

EXPOSE 80

ENTRYPOINT ["/usr/sbin/httpd", "-D", "FOREGROUND"]

$ docker build -t centos7/apache:v1 .
$ docker run -d --rm --name webserver -P centos7/apache:v1
$ docker container ls --format '{{.Names}} {{.Ports}}'
webserver 0.0.0.0:32769->80/tcp
#~OR~
$ docker port webserver | cut -d: -f2
32769
#~OR~
$ docker inspect webserver | jq -crM '[.[] | .NetworkSettings.Ports."80/tcp"[] | .HostPort] | .[]'
32769
$ curl localhost:32769
This is a custom index file built during the image creation
$ time docker stop webserver
real	0m0.283s
user	0m0.004s
sys	0m0.008s

Note that I passed --rm to the `docker run` command so that the container will be removed when I stop the container. Also, note how much faster the container stopped (~300ms vs. 10 seconds above).

HEALTHCHECK

The HEALTHCHECK instruction tells Docker how to test a container to ensure it is still working. This can detect cases such as a web server that is stuck in an infinite loop and unable to handle new connections, even though the server process is still running.

For example, we have a Dockerfile to define a simple webapp:

$ cat << EOF > Dockerfile
FROM nginx:1.13.1

RUN apt-get update \
    && apt-get install -y curl \
    && rm -rf /var/lib/apt/lists/*

HEALTHCHECK --interval=15s --timeout=3s \
    CMD curl -fs http://localhost:80/ || exit 1
EOF

This will check every five minutes or so that a web server is able to serve the site's main page within three seconds. The command's exit status indicates the health status of the container.

The possible values are:

• 0: success - the container is healthy and ready for use
• 1: unhealthy - the container is not working correctly
• 2: reserved - do not use this exit code

Then use Docker to build an image:

$ docker build -t healthcheck:v1 .

And run a container using this image:

$ docker run -d --name healthcheck-demo -p 80:80 healthcheck:v1

Then, check the status of the container:

$ docker ps
CONTAINER ID        IMAGE               COMMAND                  CREATED             STATUS                            PORTS                NAMES
9d8cf6d698c7        healthcheck:v1      "nginx -g 'daemon ..."   3 seconds ago       Up 2 seconds (health: starting)   0.0.0.0:80->80/tcp   healthcheck-demo

At the beginning, the status of container is (health: starting); after a while, it changes to be healthy:

$ docker ps                        
CONTAINER ID        IMAGE               COMMAND                  CREATED             STATUS                   PORTS                NAMES
9d8cf6d698c7        healthcheck:v1      "nginx -g 'daemon ..."   2 minutes ago       Up 5 minutes (healthy)   0.0.0.0:80->80/tcp   healthcheck-demo

It takes retries/consecutive failures of the health check for the container to be considered unhealthy.

You can use your own script to replace the command curl -fs http://localhost:80/ || exit 1.

What is more, STDOUT and STDERR of your script can be fetched from docker inspect command:

$ docker inspect --format '{{json .State.Health}}' healthcheck-demo | python -m json.tool
{
    "FailingStreak": 0,
    "Log": [
        {
            "End": "2023-03-02T19:39:58.379906565+08:00",
            "ExitCode": 0,
            "Output": "  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current\n                                 Dload  Upload   Total   Spent    Left  Speed\n\r  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0\r100   612  100   612    0     0  97297      0 --:--:-- --:--:-- --:--:--   99k\n<!DOCTYPE html>\n<html>\n<head>\n<title>Welcome to nginx!</title>\n<style>\n    body {\n        width: 35em;\n        margin: 0 auto;\n        font-family: Tahoma, Verdana, Arial, sans-serif;\n    }\n</style>\n</head>\n<body>\n<h1>Welcome to nginx!</h1>\n<p>If you see this page, the nginx web server is successfully installed and\nworking. Further configuration is required.</p>\n\n<p>For online documentation and support please refer to\n<a href=\"http://nginx.org/\">nginx.org</a>.<br/>\nCommercial support is available at\n<a href=\"http://nginx.com/\">nginx.com</a>.</p>\n\n<p><em>Thank you for using nginx.</em></p>\n</body>\n</html>\n",
            "Start": "2023-03-02T19:39:58.229550952+08:00"
        }
    ],
    "Status": "healthy"
}

Note that you can filter containers by health status with:

$ docker ps -a --filter=health=unhealthy

Or, even:

$ docker ps -a -f status=dead

Container volume management

$ docker run -it --name voltest -v /mydata centos:latest /bin/bash
[root@bffdcb88c485 /]# df -h
Filesystem                   Size  Used Avail Use% Mounted on
none                         213G  173G   30G  86% /
tmpfs                        7.8G     0  7.8G   0% /dev
tmpfs                        7.8G     0  7.8G   0% /sys/fs/cgroup
/dev/mapper/ubuntu--vg-root  213G  173G   30G  86% /mydata
shm                           64M     0   64M   0% /dev/shm
tmpfs                        7.8G     0  7.8G   0% /sys/firmware
[root@bffdcb88c485 /]# echo "testing" >/mydata/mytext.txt
$ docker inspect voltest | jq -crM '.[] | .Mounts[].Source'
/var/lib/docker/volumes/2a53fd295595690200a63def8a333b54682174923339130d560fb77ecbe41a3b/_data
$ sudo cat /var/lib/docker/volumes/2a53fd295595690200a63def8a333b54682174923339130d560fb77ecbe41a3b/_data/mytext.txt
testing
$ sudo /bin/bash -c \
  "echo 'this is from the host OS' >/var/lib/docker/volumes/2a53fd295595690200a63def8a333b54682174923339130d560fb77ecbe41a3b/_data/host.txt"
[root@bffdcb88c485 /]# cat /mydata/host.txt 
this is from the host OS

• Cleanup

$ docker rm voltest
$ docker volume rm 2a53fd295595690200a63def8a333b54682174923339130d560fb77ecbe41a3b

• Mount host's current working directory inside container:

$ echo "my config" >my.conf
$ echo "my message" >message.txt
$ echo "aerwr3adf" >app.bin
$ chmod +x app.bin
$ docker run -it --name voltest -v ${PWD}:/mydata centos:latest /bin/bash
[root@f5f34ccb54fb /]# ls -l /mydata/
total 24
-rwxrwxr-x 1 1000 1000 10 Mar  8 19:29 app.bin
-rw-rw-r-- 1 1000 1000 11 Mar  8 19:29 message.txt
-rw-rw-r-- 1 1000 1000 10 Mar  8 19:29 my.conf
[root@f5f34ccb54fb /]# touch /mydata/foobar
$ ls -l ${PWD}
total 24
-rwxrwxr-x 1 xtof  xtof  10 Mar  8 11:29 app.bin
-rw-r--r-- 1 root  root   0 Mar  8 11:36 foobar
-rw-rw-r-- 1 xtof  xtof  11 Mar  8 11:29 message.txt
-rw-rw-r-- 1 xtof  xtof  10 Mar  8 11:29 my.conf
$ docker rm voltest

Images

Saving and loading images

$ docker pull centos:latest
$ docker run -it centos:latest /bin/bash
[root@29fad368048c /]# yum update -y
[root@29fad368048c /]# echo xtof >/root/built_by.txt
$ docker commit reverent_elion centos:xtof
$ docker rm reverent_elion
$ docker images
REPOSITORY   TAG      IMAGE ID       CREATED         SIZE
centos       xtof     e0c8bd35ba50   3 seconds ago   463MB
centos       latest   980e0e4c79ec   1 minute ago    197MB
$ docker history centos:xtof
IMAGE          CREATED             CREATED BY                                      SIZE
e0c8bd35ba50   27 seconds ago      /bin/bash                                       266MB               
980e0e4c79ec   18 months ago       /bin/sh -c #(nop)  CMD ["/bin/bash"]            0B                  
<missing>      18 months ago       /bin/sh -c #(nop)  LABEL name=CentOS Base ...   0B                  
<missing>      18 months ago       /bin/sh -c #(nop) ADD file:e336b45186086f7...   197MB               
<missing>      18 months ago       /bin/sh -c #(nop)  MAINTAINER https://gith...   0B

• Save the original centos:latest image we pulled from Docker Hub:

$ docker save --output centos-latest.tar centos:latest

Note that the above command essentially tars up the contents of the image found in /var/lib/docker/image directory.

$ tar tvf centos-latest.tar 
-rw-r--r-- 0/0        2309 2016-09-06 14:10 980e0e4c79ec933406e467a296ce3b86685e6b42eed2f873745e6a91d718e37a.json
drwxr-xr-x 0/0           0 2016-09-06 14:10 ad96ed303040e4a7d1ee0596bb83db3175388259097dee50ac4aaae34e90c253/
-rw-r--r-- 0/0           3 2016-09-06 14:10 ad96ed303040e4a7d1ee0596bb83db3175388259097dee50ac4aaae34e90c253/VERSION
-rw-r--r-- 0/0        1391 2016-09-06 14:10 ad96ed303040e4a7d1ee0596bb83db3175388259097dee50ac4aaae34e90c253/json
-rw-r--r-- 0/0   204305920 2016-09-06 14:10 ad96ed303040e4a7d1ee0596bb83db3175388259097dee50ac4aaae34e90c253/layer.tar
-rw-r--r-- 0/0         202 1969-12-31 16:00 manifest.json
-rw-r--r-- 0/0          89 1969-12-31 16:00 repositories

• Save space by compressing the tar file:

$ gzip centos-latest.tar  # .tar -> 195M; .tar.gz -> 68M

• Delete the original centos:latest image:

$ docker rmi centos:latest

• Restore (or load) the image back to our local repository:

$ docker load --input centos-latest.tar.gz

Tagging images

• List our current images:

$ docker images
REPOSITORY    TAG    IMAGE ID            CREATED             SIZE
centos        xtof   e0c8bd35ba50        About an hour ago   463MB

• Tag the above image:

$ docker tag e0c8bd35ba50 xtof/centos:v1
$ docker images
REPOSITORY    TAG    IMAGE ID            CREATED             SIZE
centos        xtof   e0c8bd35ba50        About an hour ago   463MB
xtof/centos   v1     e0c8bd35ba50        About an hour ago   463MB

Note that we did not create a new image, we just created a new tag of the same/original centos:xtof image.

Note: The maximum number of characters in a tag is 128.

Docker networking

Default networks

$ ip addr show docker0
4: docker0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default 
    link/ether 02:42:c0:75:70:13 brd ff:ff:ff:ff:ff:ff
    inet 172.17.0.1/16 scope global docker0
       valid_lft forever preferred_lft forever
    inet6 fe80::42:c0ff:fe75:7013/64 scope link 
       valid_lft forever preferred_lft forever
#~OR~
$ ifconfig docker0
docker0   Link encap:Ethernet  HWaddr 02:42:c0:75:70:13  
          inet addr:172.17.0.1 Bcast:0.0.0.0  Mask:255.255.0.0
          inet6 addr: fe80::42:c0ff:fe75:7013/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:420654 errors:0 dropped:0 overruns:0 frame:0
          TX packets:1162975 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0 
          RX bytes:85851647 (85.8 MB)  TX bytes:1196235716 (1.1 GB)

$ docker network inspect bridge | jq '.[] | .IPAM.Config[].Subnet'
"172.17.0.0/16"

So, the usable range of IP addresses in our 172.17.0.0/16 subnet is: 172.17.0.1 - 172.17.255.254

$ docker network ls
NETWORK ID          NAME                DRIVER              SCOPE
bf831059febc        bridge              bridge              local
266f6df5c44e        host                host                local
ce79e4043a20        none                null                local
$ docker ps -q | wc -l
#~OR~
$ docker container ls --format '{{.Names}}' | wc -l
4  # => 4 running containers
$ docker network inspect bridge | jq '.[] | .Containers[].IPv4Address'
"172.17.0.2/16"
"172.17.0.5/16"
"172.17.0.4/16"
"172.17.0.3/16"

The output from the last command are the IP addresses of the 4 containers currently running on my host.

Custom networks

• Create a Docker network

$ man docker-network-create  # for details
$ docker network create --subnet 10.1.0.0/16 --gateway 10.1.0.1 --ip-range=10.1.4.0/24 \
    --driver=bridge --label=host4network br04

• Use the above network with a given container:

$ docker run -it --name net-test --net br04 centos:latest /bin/bash

• Assign a static IP to a given container in the above (user created) network:

$ docker run -it --name net-test --net br04 --ip 10.1.4.100 centos:latest /bin/bash

Note: You can only assign static IPs to user created networks (i.e., you cannot assign them to the default "bridge" network).

Monitoring

$ docker top <container_name>
$ docker stats <container_name>

Logs

• Fetch logs of a given container:

$ docker logs <container_name>

• Fetch logs of a given container prefixed with timestamps (UTC format by default):

$ docker logs --timestamps <container_name>

Events

$ docker events
$ docker events --since '1h'
$ docker events --since '2018-03-08T16:00'
$ docker events --filter event=attach
$ docker events --filter event=destroy
$ docker events --filter event=attach --filter event=die --filter event=stop

Cleanup

• Check local system disk usage:

$ docker system df
TYPE                TOTAL               ACTIVE              SIZE                RECLAIMABLE
Images              53                  3                   16.52GB             15.9GB (96%)
Containers          3                   1                   438.9MB             0B (0%)
Local Volumes       16                  2                   2.757GB             2.628GB (95%)
Build Cache         0                   0                   0B                  0B

Note: Use docker system df --verbose to get even more details.

• Delete all stopped containers at once and reclaim the disk space they are using:

$ docker container prune

• Remove all containers (both the running ones and the stopped ones):

# Old method:
$ docker rm -f $(docker ps -aq)
# New method:
$ docker container rm -f $(docker container ls -aq)

Note: It is often useful to use the --rm flag when running a container so that it is automatically removed when its PID 1 process is stopped, thus releasing unused disk immediately.

• Cleanup everything all at one (CAREFUL!):

$ docker system prune
WARNING! This will remove:
  - all stopped containers
  - all networks not used by at least one container
  - all dangling images
  - all dangling build cache
Are you sure you want to continue? [y/N]

Examples

Simple Nginx server

• Create an index.html file:

$ mkdir html
$ cat << EOF >html/index.html
Hello from Docker
EOF

• Create a Dockerfile:

FROM nginx
COPY html /usr/share/nginx/html

• Build the image:

$ docker build -t test-nginx .

• Start up container, using image built above:

$ docker run --name check-nginx -d -p 8080:80 test-nginx

• Check that it works:

$ curl http://localhost:8080
Hello from Docker

Connecting two containers

In this example, we will start up a Postgres container and then start up another container and make a connection to the original Postgres container:

$ docker pull postgres
$ docker run --name test-postgres -e POSTGRES_PASSWORD=mypassword -d postgres
$ docker run -it --rm --link test-postgres:postgres postgres psql -h postgres -U postgres

Password for user postgres:
psql (11.0 (Debian 11.0-1.pgdg90+2))
Type "help" for help.

postgres=# SELECT 1;
 ?column?
----------
        1
(1 row)

postgres=# \q

Connection was successful!

Support for various hardware platforms

NOTE: If your image is being created on an M1 chip (ARM64) but you want to execute the container on an AMD64 chip, then use FROM - platform=linux/amd64 on your Docker image so it can be shipped anywhere. For example:

FROM node:current-alpine3.15
#FROM - platform=linux/amd64 node:current-alpine3.15
WORKDIR /app
ADD . /app
RUN npm install
#RUN npm install express
EXPOSE 3000
CMD ["npm", "start"]

Docker compose

Compose is a tool for defining and running multi-container Docker applications. With Compose, you use a YAML file to configure your application's services. Then, with a single command, you create and start all the services from your configuration. To learn more about all the features of Compose, see the list of features.

Using Compose is basically a three-step process:

1. Define your app's environment with a Dockerfile. so it can be reproduced anywhere.
2. Define the services that make up your app in docker-compose.yml so they can be run together in an isolated environment.
3. Run docker-compose up and Compose starts and runs your entire app.

Basic example

Note: This is based off of this article.

In this basic example, we will build a simple Python web application running on Docker Compose. The application uses the Flask framework and maintains a hit counter in Redis.

Note: This section assumes you already have Docker Engine and Docker Compose installed.

• Create a directory for the project:

$ mkdir compose-test && cd $_

• Create a file called app.py in your project directory and paste this in:

import time
import redis
from flask import Flask


app = Flask(__name__)
cache = redis.Redis(host='redis', port=6379)


def get_hit_count():
    retries = 5
    while True:
        try:
            return cache.incr('hits')
        except redis.exceptions.ConnectionError as exc:
            if retries == 0:
                raise exc
            retries -= 1
            time.sleep(0.5)


@app.route('/')
def hello():
    count = get_hit_count()
    return 'Hello World! I have been seen {} times.\n'.format(count)

if __name__ == "__main__":
    app.run(host="0.0.0.0", debug=True)

In this example, redis is the hostname of the redis container on the application's network. We use the default port for Redis: 6379.

• Create another file called requirements.txt in your project directory and paste this in:

flask
redis

• Create a Dockerfile

  This Dockerfile will be used to build an image that contains all the dependencies the Python application requires, including Python itself.

FROM python:3.4-alpine
ADD . /code
WORKDIR /code
RUN pip install -r requirements.txt
CMD ["python", "app.py"]

• Create a file called docker-compose.yml in your project directory and paste the following:

version: '3'
services:
  web:
    build: .
    ports:
     - "5000:5000"
  redis:
    image: "redis:alpine"

• Build and run this app with Docker Compose:

$ docker-compose up

Compose pulls a Redis image, builds an image for your code, and starts the services you defined. In this case, the code is statically copied into the image at build time.

• Test the application:

$ curl localhost:5000
Hello World! I have been seen 1 times.

$ for i in $(seq 1 10); do curl -s localhost:5000; done
Hello World! I have been seen 2 times.
Hello World! I have been seen 3 times.
Hello World! I have been seen 4 times.
Hello World! I have been seen 5 times.
Hello World! I have been seen 6 times.
Hello World! I have been seen 7 times.
Hello World! I have been seen 8 times.
Hello World! I have been seen 9 times.
Hello World! I have been seen 10 times.
Hello World! I have been seen 11 times.

• List containers:

$ docker-compose ps
       Name                      Command               State           Ports          
-------------------------------------------------------------------------------------
compose-test_redis_1   docker-entrypoint.sh redis ...   Up      6379/tcp               
compose-test_web_1     python app.py                    Up      0.0.0.0:5000->5000/tcp

• Display the running processes:

$ docker-compose top
compose-test_redis_1
  UID       PID    PPID    C   STIME   TTY     TIME         CMD      
--------------------------------------------------------------------
systemd+   29401   29367   0   15:28   ?     00:00:00   redis-server 

compose-test_web_1
UID     PID    PPID    C   STIME   TTY     TIME                 CMD              
--------------------------------------------------------------------------------
root   29407   29373   0   15:28   ?     00:00:00   python app.py                
root   29545   29407   0   15:28   ?     00:00:00   /usr/local/bin/python app.py

• Shutdown app:

$ Ctrl+C
#~OR~
$ docker-compose down

Install docker

Debian-based distros

Ubuntu 16.04 (Xenial Xerus)

Note: For this install, I will be using Ubuntu 16.04 LTS (Xenial Xerus). Docker requires a 64-bit version of Ubuntu as well as a kernel version equal to or greater than 3.10. My system satisfies both requirements.

• Setup the docker repo to install from:

$ sudo apt-get update -y
$ sudo apt-key adv --keyserver hkp://p80.pool.sks-keyservers.net:80 --recv-keys 58118E89F3A912897C070ADBF76221572C52609D
$ echo "deb https://apt.dockerproject.org/repo ubuntu-xenial main" | sudo tee /etc/apt/sources.list.d/docker.list
$ sudo apt-get update -y

Make sure you are about to install from the Docker repo instead of the default Ubuntu 16.04 repo:

$ apt-cache policy docker-engine

The output of the above command show look something like the following:

docker-engine:
  Installed: (none)
  Candidate: 17.05.0~ce-0~ubuntu-xenial
  Version table:
     17.05.0~ce-0~ubuntu-xenial 500
        500 https://apt.dockerproject.org/repo ubuntu-xenial/main amd64 Packages
     17.04.0~ce-0~ubuntu-xenial 500
        500 https://apt.dockerproject.org/repo ubuntu-xenial/main amd64 Packages
...

• Install docker:

$ sudo apt-get install -y docker-engine

Ubuntu 18.04 (Bionic Beaver)

$ sudo apt update
$ sudo apt install -y apt-transport-https ca-certificates curl software-properties-common
$ curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo apt-key add -
$ sudo add-apt-repository "deb [arch=amd64] https://download.docker.com/linux/ubuntu $(lsb_release -cs) stable"
$ sudo apt update
$ apt-cache policy docker-ce

docker-ce:
  Installed: (none)
  Candidate: 5:18.09.0~3-0~ubuntu-bionic
  Version table:
     5:18.09.0~3-0~ubuntu-bionic 500
        500 https://download.docker.com/linux/ubuntu bionic/stable amd64 Packages

$ sudo apt install docker-ce -y
$ sudo systemctl status docker

● docker.service - Docker Application Container Engine
   Loaded: loaded (/lib/systemd/system/docker.service; enabled; vendor preset: enabled)
   Active: active (running) since Tue 2018-12-04 13:40:36 PST; 4s ago
     Docs: https://docs.docker.com
 Main PID: 6134 (dockerd)
    Tasks: 16
   CGroup: /system.slice/docker.service
           └─6134 /usr/bin/dockerd -H unix://

Red Hat-based distros

Note: For this install, I will be using CentOS 7 (release 7.2.1511). Docker requires a 64-bit version of CentOS as well as a kernel version equal to or greater than 3.10. My system satisfies both requirements.

• Install Docker (the fast way):

$ sudo yum update -y
$ curl -fsSL https://get.docker.com/ | sh

• Install Docker (via a yum repo):

$ sudo yum update -y
$ sudo pip install docker-py
$ cat << EOF > /etc/yum.repos.d/docker.repo
[dockerrepo]
name=Docker Repository
baseurl=https://yum.dockerproject.org/repo/main/centos/7/
enabled=1
gpgcheck=1
gpgkey=https://yum.dockerproject.org/gpg
EOF
$ sudo rpm -vv --import https://yum.dockerproject.org/gpg
$ sudo yum update -y
$ sudo yum install docker-engine -y

Post-installation steps

• Check on the status of docker:

$ sudo systemctl status docker

● docker.service - Docker Application Container Engine
   Loaded: loaded (/lib/systemd/system/docker.service; enabled; vendor preset: enabled)
   Active: active (running) since Tue 2016-07-12 12:31:08 PDT; 6s ago
     Docs: https://docs.docker.com
 Main PID: 3392 (docker)
   CGroup: /system.slice/docker.service
           ├─3392 /usr/bin/docker daemon -H fd://
           └─3411 docker-containerd -l /var/run/docker/libcontainerd/docker-containerd.sock --runtime docker-runc --start-timeout 2m

• Make sure the docker service automatically starts after a machine reboot:

$ sudo systemctl enable docker

• Execute docker without `sudo`:

$ sudo usermod -aG docker $(whoami)
#~OR~
$ sudo usermod -aG docker $USER

Log out and log back in to use docker without `sudo`.

• Check version of Docker installed:

$ docker version
Client:
 Version:      17.05.0-ce
 API version:  1.29
 Go version:   go1.7.5
 Git commit:   89658be
 Built:        Thu May  4 22:10:54 2017
 OS/Arch:      linux/amd64

Server:
 Version:      17.05.0-ce
 API version:  1.29 (minimum version 1.12)
 Go version:   go1.7.5
 Git commit:   89658be
 Built:        Thu May  4 22:10:54 2017
 OS/Arch:      linux/amd64
 Experimental: false

• Check that docker has been successfully installed and configured:

$ docker run hello-world

...
This message shows that your installation appears to be working correctly.
...

As the above message shows, you now have a successful install of Docker on your machine and are ready to start building images and creating containers.

Miscellaneous

• Get the hostname of the host the Docker Engine is running on:

$ docker info -f '{{ .Name }}'

• Get the number of stopped containers:

$ docker info --format '{{json .}}' | jq '.ContainersStopped'
3

• Get the number of images in the local registry:

$ docker info --format '{{json .}}' | jq '.Images'
92

• Verify the Docker service is running:

$ curl -H "Content-Type: application/json" --unix-socket /var/run/docker.sock http://localhost/_ping
OK

• Show docker disk usage

$ docker system df
TYPE            TOTAL     ACTIVE    SIZE      RECLAIMABLE
Images          84        11        25.01GB   20.44GB (81%)
Containers      20        0         768.1MB   768.1MB (100%)
Local Volumes   16        2         2.693GB   2.628GB (97%)
Build Cache     0         0         0B        0B

• Just just the version of Docker installed:

$ docker version --format '{{.Server.Version}}'
20.10.7
$ docker version --format '{{.Server.Version}}' 2>/dev/null || docker -v | awk '{gsub(/,/, "", $3); print $3}'
20.10.7

Install your own Docker private registry

Note: I will use CentOS 7 for this install and assume you already have docker and docker-compose installed (see above).

For this install, I will assume you have a domain name registered somewhere. I will use docker.example.com as my example domain. Replace anywhere you see that below with your actual domain name.

• Install dependencies:

$ yum install -y nginx  # used for the registry endpoint
$ yum install -y httpd-tools  # for the htpasswd utility

• Setup docker registry directory structure:

$ mkdir -p /opt/docker-registry/{data,nginx{/conf.d,/certs},log}
$ cd /opt/docker-registry

• Create a docker-compose file:

$ vim docker-compose.yml  # and add the following:

nginx:
  image: "nginx:1.9"
  ports:
    - 5043:443
  links:
    - registry:registry
  volumes:
    - ./log/nginx/:/var/log/nginx:rw
    - ./nginx/conf.d:/etc/nginx/conf.d:ro
    - ./nginx/certs:/etc/nginx/certs:ro
registry:
  image: registry:2
  ports:
    - 127.0.0.1:5000:5000
  environment:
    REGISTRY_STORAGE_FILESYSTEM_ROOTDIRECTORY: /data
  volumes:
    - ./data:/data

• Create an Nginx configuration file:

$ vim /opt/docker-registry/nginx/conf.d/registry.conf  # and add the following:

upstream docker-registry {
  server registry:5000;
}

server {
  listen 443;
  server_name docker.example.com;

  # SSL
  ssl on;
  ssl_certificate /etc/nginx/certs/docker.example.com.crt;
  ssl_certificate_key /etc/nginx/certs/docker.example.com.key;

  # disable any limits to avoid HTTP 413 for large image uploads
  client_max_body_size 0;

  # required to avoid HTTP 411: see Issue #1486 (https://github.com/docker/docker/issues/1486)
  chunked_transfer_encoding on;

  location /v2/ {
    # Do not allow connections from docker 1.5 and earlier
    # docker pre-1.6.0 did not properly set the user agent on ping, catch "Go *" user agents
    if ($http_user_agent ~ "^(docker\/1\.(3|4|5(?!\.[0-9]-dev))|Go ).*$" ) {
      return 404;
    }

    proxy_pass                          http://docker-registry;
    proxy_set_header  Host              $http_host;   # required for docker client's sake
    proxy_set_header  X-Real-IP         $remote_addr; # pass on real client's IP
    proxy_set_header  X-Forwarded-For   $proxy_add_x_forwarded_for;
    proxy_set_header  X-Forwarded-Proto $scheme;
    proxy_read_timeout                  900;

    add_header 'Docker-Distribution-Api-Version:' 'registry/2.0' always;

    # To add basic authentication to v2 use auth_basic setting plus add_header
    auth_basic "Restricted access to Docker Registry";
    auth_basic_user_file /etc/nginx/conf.d/registry.htpasswd;
  }
}

$ cd /opt/docker-registry/nginx/conf.d
$ htpasswd -c registry.htpasswd <username>  # replace <username> with your actual username
$ htpasswd registry.htpasswd <username2>  # [optional] add a 2nd user

• Setup your own certificate signing authority (for use with SSL):

$ cd /opt/docker-registry/nginx/certs

• Generate a new root key:

$ openssl genrsa -out docker-registry-CA.key 2048

• Generate a root certificate (enter anything you like at the prompts):

$ openssl req -x509 -new -nodes -key docker-registry-CA.key -days 3650 -out docker-registry-CA.crt

Then generate a key for your server (this is the file referenced by ssl_certificate_key in the Nginx configuration above):

$ openssl genrsa -out docker.example.com.key 2048

Now we have to make a certificate signing request (CSR). After you type the following command, OpenSSL will prompt you to answer a few questions. Enter anything you like for the first few, however, when OpenSSL prompts you to enter the "Common Name", make sure to enter the domain or IP of your server.

$ openssl req -new -key docker.example.com.key -out docker.example.com.csr

• Sign the certificate request:

$ openssl x509 -req -in docker.example.com.csr -CA docker-registry-CA.crt -CAkey docker-registry-CA.key -CAcreateserial -out docker.example.com.crt -days 3650

• Force any clients that will use the certificate authority we created above to accept that it is a "legitimate" certificate. Run the following commands on the Docker registry server and on any hosts that will be communicating with the Docker registry server:

$ sudo cp /opt/docker-registry/nginx/certs/docker-registry-CA.crt /usr/local/share/ca-certificates/
$ sudo update-ca-trust

• Restart the Docker daemon in order for it to pick up the changes to the certificate store:

$ sudo systemctl restart docker.service

• Bring up the associated Docker containers:

$ docker-compose up -d

• Your Docker registry directory structure should look like the following:

$ cd /opt/docker-registry && tree .
.
├── data
├── docker-compose.yml
├── log
│   └── nginx
│       ├── access.log
│       └── error.log
└── nginx
    ├── certs
    │   ├── docker-registry-CA.crt
    │   ├── docker-registry-CA.key
    │   ├── docker-registry-CA.srl
    │   ├── docker.example.com.crt
    │   ├── docker.example.com.csr
    │   └── docker.example.com.key
    └── conf.d
        ├── registry.conf
        └── registry.htpasswd

• To access the private Docker registry from a client machine (any machine, really), first add the SSL certificate you created earlier to the client machine:

$ cat /opt/docker-registry/nginx/certs/docker-registry-CA.crt  # copy contents
# On client machine:
$ sudo vim /usr/local/share/ca-certificates/docker-registry-CA.crt  # paste contents
$ sudo update-ca-certificates  # You should see "1 added" in the output

• Restart Docker on the client machine to make sure it reloads the system's CA certificates:

$ sudo service docker restart

• Test that you can reach your private Docker registry:

$ curl -k https://USERNAME:PASSWORD@docker.example.com:5043/v2/
{}  # <- proper output

• Now, test that you can login with Docker:

$ docker login https://docker.example.com:5043

If that returns with "Login Succeeded", your private Docker registry is up and running!

This section is incomplete. It will be updated when I have time.

Docker environment variables

Note: See here for the most up-to-date list of environment variables.

The following list of environment variables are supported by the docker command line:

DOCKER_API_VERSION 

The API version to use (e.g., 1.19)

DOCKER_CONFIG 

The location of your client configuration files.

DOCKER_CERT_PATH 

The location of your authentication keys.

DOCKER_DRIVER 

The graph driver to use.

DOCKER_HOST 

Daemon socket to connect to.

DOCKER_NOWARN_KERNEL_VERSION 

Prevent warnings that your Linux kernel is unsuitable for Docker.

DOCKER_RAMDISK 

If set this will disable "pivot_root".

DOCKER_TLS_VERIFY 

When set Docker uses TLS and verifies the remote.

DOCKER_CONTENT_TRUST 

When set Docker uses notary to sign and verify images. Equates to --disable-content-trust=false for build, create, pull, push, run.

DOCKER_CONTENT_TRUST_SERVER 

The URL of the Notary server to use. This defaults to the same URL as the registry.

DOCKER_TMPDIR 

Location for temporary Docker files.

Because Docker is developed using "Go", one can also use any environment variables used by the "Go" runtime. In particular, the following might be useful:

HTTP_PROXY

HTTPS_PROXY

NO_PROXY

• Example usage:

$ export DOCKER_API_VERSION=1.19

See also

• containerd

References

1. ↑ https://www.docker.com/what-docker

External links

• Official website