class: center, middle # Docker for Developers ## Part 1 --- # Agenda What you should probably know before this talk - bash - minimally, enough to run commands and navigate the filesystem - What a virtual machine is - Some basic networking - Some (very) basic network programming What (I hope) this talk teaches you - What Docker is - What a container is - How containers differ from VMs - conceptually, don't expect a libcontainer deep dive - How VMs and Containers co-exist - How to run a container - How to use containers to help you write your code - How to containerize your own code - How to run your containerized code ??? The agenda for todays talk Things you should know: bash, of VMs, networking and programs that communicate over networks More knowledge of bash is handy for more complicated flags/args/etc, but traversing the file system is pretty important for understanding examples, when it comes to building images We're going to learn: - What Docker is - What a container is - How containers differ from VMs - How VMs are containers coexist, and why By the end of the talk you will have: - Run a container - Run a service within a container - Used a container to help with development - Containized and run a project --- # What is Docker? > Docker is the world’s leading software container platform. Developers use Docker to eliminate “works on my machine” problems when collaborating on code with co-workers. Operators use Docker to run and manage apps side-by-side in isolated containers to get better compute density. Enterprises use Docker to build agile software delivery pipelines to ship new features faster, more securely and with confidence for both Linux, Windows Server, and Linux-on-mainframe apps. .footnote[https://www.docker.com/what-docker] ??? What is Docker? this blurb is straight off the website Kind of sales-pitchy, but gets the point across Docker is a container "platform" - the technology and tools to build, run, and manage containers and container architectures. --- # What is a Container? > Containers are a way to package software in a format that can run isolated on a shared operating system. Unlike VMs, containers do not bundle a full operating system - only libraries and settings required to make the software work are needed. This makes for efficient, lightweight, self-contained systems and guarantees that software will always run the same, regardless of where it’s deployed. .footnote[https://www.docker.com/what-docker] ??? What is a container? Straight off the website again - Not VMs - isolated (somewhat) - more info coming up next - small, fast, effecient - **repeatability** - Lend themselves very well to automated orchestration --- # Containers != VMs To blatantly steal a great metaphor (watch the talk linked below): VMs package an **entire system** - Kernel - Drivers - Libraries - Software - etc VMs are like houses. - Houses are freestanding buildings - Houses have their own plumbing, electrical, etc .footnote[[Mike Coleman, "Docker?!? But I'm a SYSADMIN!" - Dockercon 2017](https://www.youtube.com/watch?v=M7ZBF-JJWVU)] ??? Containers aren't VMs, VMs aren't containers Mike Coleman, a presenter at Dockercon, came up with a great metaphor House:Apartment::VM:Container VMs package _everything_, they are houses. Independent, don't share resources, etc --- # Containers != VMs, cont'd Containers package a **service** - Libraries - Application Containers are like apartments - There are many apartments in a single building - Apartments share access to many key components, eg: plumbling, electrical .footnote[[Mike Coleman, "Docker?!? But I'm a SYSADMIN!" - Dockercon 2017](https://www.youtube.com/watch?v=M7ZBF-JJWVU)] ??? Containers package as little as possible. Share resources, but are safely isolated Containers are like apartments. You and your neighbors share pipes, but you don't want to know when they flush. --- # Containers != VMs, cont'd With this in mind, VMs and containers are typically used in different situations, with different expectations. - VMs are commonly used to package complex, potentially stateful, applications. - VMs are more "permanent" after being created. - VMs more frequently run multiple services per VM <hr /> - Containers are commonly used to package simple, usually stateless, applications. - Containers are ephemeral, appearing and disappearing as required - Containers often run a single service (if not single process) per container ??? VMs and Containers are typically used in different situations VMs == old guard, workhorses. Still definitely good for certain applications, long term, _highly_ customized, whole "appliances" Containers solve problems devs and ops were experiencing. Turn code into a shippable unit Highly scalable, informed by/informing move towards microservices Orchestration as code. DevOps --- # Containers ∪ VMs Containers and Virtual Machines are different. Containers and VMs are both appropriate in different situations. You should probably know how to use both containers and virtual machines. You should also know when to use containers and/or virtual machines. (hint: the talk below helps know when to use what, watch it) .footnote[[Mike Coleman, "Docker?!? But I'm a SYSADMIN!" - Dockercon 2017](https://www.youtube.com/watch?v=M7ZBF-JJWVU)] ??? Despite their differences, VMs and containers can (and probably must) live in harmony The container-folk aren't coming for your VMs with torches and pitchforks Often the most appropriate orchestrations involve VMs **and** containers Sometimes the containers are even running on VMs [gasp] But this isn't a talk about VMs, so lets get on with it --- # Demo Preparation We're about to do some demos. - Have docker installed? - Cool - you can do them locally if you want to - Don't have docker installed? (or just don't want to do them locally?) - Also cool - you can use [play-with-docker.com](http://play-with-docker.com) If you're using play-with-docker, go ahead and click "+ Add New Instance" on the left. **Note**: Can't see "+ Add New Instance"? The website gets a little weird with wrapping sometimes, try and make it fullscreen. ??? Time for some demos Best way to learn is to get your hands dirty Give people a minute to get setup Encourage people to help those next to them having trouble --- # Running a Container So, now that we know (more or less) what containers are, and we've got ourselves setup with play-with-docker (or our local docker environments)... Let's try running one! This will run an Ubuntu container, and drop you to a bash prompt ```bash $ docker run -t -i ubuntu bash ``` If bash isn't your thing... ```bash $ docker run -ti python ``` ??? Time to run our first container. Does that say root in the prompt? Yes. Can potentially do something drastic, like delete bin. Demonstrates that **this isn't the host** Can demonstrate re-running, to show how fast it is when you don't have to download the image. Serious difference from VMs. --- # Running a Container, cont'd Command Breakdown - .remark-inline-code[docker]: The docker CLI utility - .remark-inline-code[run]: The subparser for running containers - .remark-inline-code[-t]: "tty", allocate a pseudo-tty - .remark-inline-code[-i]: "interactive", capture stdin - .remark-inline-code[ubuntu||python]: the image to use - .remark-inline-code[bash]: The command to call within the container - Some images have implicit "entrypoints", like Python does with the REPL ??? Lets dissect this command, it has a lot of information jammed in there - **Interactive containers are not necessarily the norm** - we're going to run a service next --- # Intermezzo ## What did I just download?! You just downloaded an Ubuntu (or python) docker image! - They can be pushed (or pulled) from the [DockerHub](https://hub.docker.com) - The DockerHub is a little like a package manager - Some packages, like Ubuntu or Python are "official" packages - Maintained by organization or Docker - Don't run code you don't trust. - Docker supports various security metrics for this - They are outside of the scope of this talk - Don't run programs you download off the internet - _sarcasm_ (kind of) ??? Downloaded image from Dockerhub Dockerhub is like a package manager Might better be described as something like pypi - if you're a Python person NOTE: The security note here is a valid one, but definitely outside of the scope of this talk, be sure to re-iterate to examine images occasionally (especially unofficial images, and especially if they are going to be in contact with sensitive data) --- # Running a Container, cont'd Lets try running something more "service-y" ```bash $ docker run -ti alpine ping www.google.com ``` Though the output may look identical, this ping is running from the container, not the host. ??? Lets run something more service-y. Services run continuously, perform a function (either continously or on demand) Ping makes an acceptable stand in for a fully automated service. We could run this with just -t and no -i, but then ^C just disconnects us from the container, and doesn't kill the ping. This behavior could be desirable or not, keep thoughts like this in mind when running your containers --- # Intermezzo ## What's alpine? Alpine is a tiny Linux distribution. We use it here because it's small, and so quick to download and run. Size is often a concern with Docker containers, if you are going to distribute them widely or duplicate them tens, hundreds, or thousands of times across hosts. For this reason you often see Docker images built off of an alpine base, it has some nice tools like a package manager, most of the gnu utilities, etc but is still very small compared to your typical distros (Ubuntu/Redhat/Debian/etc). This makes it ideal for maintaining core functionality and convenience in Docker images, while not bogging down images. In addition to size, it is often advisable to be running as little as possible (besides your application of course) inside of a container. A minimal distro, with fewer bells and whistles to begin with, is often easier to achieve this with. ??? Alpine is a small linux distribution Good for distribution, in case of duplication. Lots of docker images use Alpine. Alpine doesn't ship with a lot of other stuff that has to be "turned off" **Fun Fact**: If you're using play-with-docker you're on Alpine right now! --- # Running a Container, cont'd Running a service in a container isn't tremendously convenient if it locks up our terminal, however. Why don't we try running the same "service" **d**etached from our terminal. ```bash $ docker run -d alpine ping www.google.com ``` To have a peak inside... ```bash $ docker logs $identifier ``` Or, in the style of .remark-inline-code[tail -f somelog.log] ```bash $ docker logs -f $identifier ``` ??? Often times you're running more than one container, or want your container to be out of the way, so getting comfortable with these commands to "check up" on the internals of your container is good. This is also what logging from all containers looks like, by default. In your own code remember to allow for logging to stdout --- # Running a Container, cont'd ## Check Up We've run quite a few containers now, and in fact, we are still running one! Let's take a look at the running containers. ```bash $ docker ps ``` And a look at all the containers (even the stopped/exited ones) ```bash $ docker ps -a ``` And, a handy little scriplet for dumping all the container identifiers ```bash $ docker ps -aq ``` ??? These commands are just the tip of the iceberg. We don't really have the time to go through all the administrative docker functionality in this talk, just the bare bones. Docker provides a ton of functionality for managing containers and images. --- # Running a Container, cont'd ## Clean Up Stop our ping container (we don't want to DDOS google) ```bash $ docker stop $identifier ``` Remove the container ```bash $ docker rm $identifier ``` And now let's remove all our stopped containers too, to clean up ```bash $ docker rm $(docker ps -aq) ``` ??? I find keeping a "tidy" docker workspace is helpful, you only notice you've left a mess around when you need to debug something. Note: People might be be familiar with the bashism in the final command, it executes the command inside of $() and replaces that portion of the command with the output --- # Running a Container, cont'd While pinging google is great for checking network connectivity, maybe we should demonstrate something slightly more useful. In this example, we will run a service (nginx with the default configuration) in a container, and interact with it from outside of the container via port mapping. Let's fire up our container **d**etached, and use the .remark-inline-code[-p] flag to publish port 80 in the container on port 80 of the host. We can also use the .remark-inline-code[--name] flag to give our container an easy to remember name. ```bash $ docker run -d -p 80:80 --name webserver nginx ``` After we do this, interacting with port 80 on our host will forward that traffic to port 80 inside of the container ```bash $ curl localhost:80 ``` ??? Pinging google worked as a good stand-in for a service, but lets try out the real deal now. This is also the first service we'll set up to be able to interact with outside of container-land, via port mapping. --- # Running a Container, cont'd And cleanup is quick and easy ```bash $ docker stop webserver ``` ```bash $ docker rm webserver ``` Leaving nothing after we stop the container ```bash $ curl localhost:80 ``` ??? Again, keeping a relatively clean environment can pay dividends when something goes wrong. Resetting the "state" of a supporting service is as simple as stopping one, starting another --- # Containers Alongside Your Code The same principal as the previous nginx example can be used to leverage docker containers alongside your code in order to provide services during development, without the hassle of setting up, maintaining, resetting, etc test servers for services like databases. For example, imagine if I were in the early stages of writing a [Clipboard-as-a-Service web clipboard synchronization tool...](https://github.com/bnbalsamo/webclip) ??? But, how about a real world example to demonstrate these principals in action (give brief code overview in github) --- # Intermezzo ## Setting this demo up on a play-with-docker instance ```bash $ apk add --no-cache python3 $ git clone https://github.com/bnbalsamo/webclip.git $ cd webclip $ python3 -m venv venv $ source venv/bin/activate $ pip install -r requirements_dev.txt $ python setup.py install $ webclip --help ``` ??? Time permitting pause here, if people want to setup the demo themselves. Can take the time to explan python virtual environments, demo after activation $ python is 3.x, not 2.7 --- # Containers Alongside Your Code, cont'd With this written, would you rather... 1. Mock a redis object which implements the functionality the code relies on to test 2. Use a real redis server to test in order to test out your proof of concept? Option #1 - Disconnects your code from the underlying redis implementation - Are you positive you mocked that object right? - What if the dependency interface changes? - etc - Can be more complicated than the original program! - As in this case Option #2 - Requires setting up a whole redis server! ??? How do we move forward, now that we've got some code? The unspeakable option: Don't test. Option 1: Mocking. more moving parts, more chances for a mishap Can miss "gotchas" (like r.get() returning bytes, not strs) Option 2 without Docker: - emails - resources - permissions - configuration - whose in charge of this? - Whats the default state? - Is testing going to put the resource in a weird state? - etc --- # Containers Alongside Your Code, cont'd ## Docker to the rescue! ```bash $ py.test ... boom ... $ docker run -d -p 6379:6379 redis $ py.test ... :) ... ``` ??? Docker to the rescue! Docker makes all of those go away (mostly) Easy to standardize default state via images Easy to "factory reset" - delete and make a new one Everyone can be tweaking to their hearts content, not stepping on toes --- # Containers Alongside Your Code, cont'd This process can be applied to many "dockerized" network services. [Explore the Dockerhub](https://hub.docker.com/explore/) for ideas! or Dockerize your own services! ??? Processes is applicable to lots of services Dockerhub is a tremendous resource, _lots_ of projects Guess what we're going to be demoing next? --- # Building a Container Now that we've worked on how to run containers, and how to use running containers for practical benefit while developing, let's address building our own containers! The first step to building a container is understanding, at a slightly more granular level, what a container _is_. To do this, I'll employ a good ol' OO programming metaphor: - All containers are built from images, in the way that all objects are built from classes. - The same way a single class can be used to create many objects, a single image can be used to create many containers. - In much the same way a class utilizes a constructor, Docker images use Dockerfiles. - In this sense, images can be considered "base classes" which are optionally extensible. ??? Deeper understanding of Containers, in order to build our own OOP Metaphor: - Images:Containers::Classes:Objects - "blueprints for instances" - one:many - Dockerfile::Constructor - Extensability - overloading - subclasses --- # Building a Container, cont'd Dockerfiles define a series of actions which initialize the beginning "state" of every container built off the image the Dockerfile defines. Each "action" defined in a Dockerfile produces a layer, these layers are then stacked, one on top of the other, to produce the resulting image. Let's have a look at a simple Dockerfile ```dockerfile FROM nginx:alpine COPY docroot /usr/share/nginx/html CMD nginx -g "daemon off;" ``` ??? Dockerfiles define a series of commands, which produce layers Layers go on one after the other, like lines in a setup script Image the result of all of the layers stacked up --- # Building a Container, cont'd So, what exactly is going on here? - .remark-inline-code[FROM nginx:alpine]: This line describes the base image we want to begin building our image from. - This image is also built from a Dockerfile, you can look them up through the Dockerhub. This one is [here](https://github.com/nginxinc/docker-nginx/blob/1d2e2ccae2f6e478f628f4091d8a5c36a122a157/mainline/alpine/Dockerfile). - Bonus points: Trace this image back to FROM scratch - All of the layers in the FROM image will be applied, _except_ for CMD and ENTRYPOINT layers - .remark-inline-code[COPY docroot /usr/share/nginx/html]: This line copies a directory from our Docker build environment (by default, the directory the Dockerfile is in) into the container at the specified location. - The specified location here happens to be the docroot defined by the default nginx config. - .remark-inline-code[CMD nginx -g ”daemon off;”]: This line specifies the default command which will be run when a container built from this image is started, if it isn't clobbered on the command line with a different command. - **Note**: We don't want nginx to daemonize, as once the process exits or enters the background the container will automatically exit, thus the .remark-inline-code[-g daemon off;] flag ??? The FROM line: Where we're "inheriting" from - All layers _except_ ENTRYPOINT, CMD - This probably has a Dockerfile too - "scratch" == empty image, beginning COPY == copy from **host to container** - RUN cp src dst == copies within the container CMD, probably runs the service. When CMD exits, container stops --- # Building a Container, cont'd Let's go ahead and try some experimenting with this Dockerfile To set up our environment ```bash $ mkdir my_website $ cd my_website $ nano Dockerfile # or your preferred editor $ # Write (or copy/paste) the Dockerfile $ mkdir docroot $ nano docroot/index.html # or your preferred editor $ # Put whatever you want in the index! ``` To build our image ```bash $ docker build . -t my_website ``` - .remark-inline-code[docker]: Use the docker CLI - .remark-inline-code[build]: Use the 'build' subparser - .remark-inline-code[.]: Use this directory as the build environment - Defaults to looking for a file called "Dockerfile" in the specified dir - .remark-inline-code[-t my_website]: Tag the built image "my_website" ??? Lets fire this Dockerfile up Need to make an environment The environment is important, if the Docker engine is remote it **all** gets packaged up and shipped out --- # Building a Container, cont'd Any now, we can run containers using the image we just built! Par for the course ```bash $ docker run -d -p 80:80 my_website ``` And another, bound to a different port ```bash $ docker run -d -p 8080:80 my_website ``` Some bonus commands to examine the output of ```bash $ docker ps $ curl localhost:80 $ curl localhost:8080 $ docker run -ti my_website /bin/sh ``` ??? Now that we built an image, lets run a container Woohoo! We built a container and can see our content! --- # Building a Container, cont'd And, a bonus exercise: ```bash $ docker run -d -p 80:80 -v $(pwd)/docroot:/usr/share/nginx/html nginx:alpine $ curl localhost <output> $ nano index.html # make some changes $ curl localhost <output> ``` We've introduced a new command here, .remark-inline-code[-v]. The .remark-inline-code[-v] flag tells the Docker engine to "mount" a directory on the host into the container. Utilizing a host mount, we can dynamically change what the container is seeing in real time without interrupting our process. Any workflows which utilize an auto-reloading server, or a service reading straight from disk can be altered/tested on the fly this way. ??? Keep thinking with layers, "clobbering" things with mounts is cool in dev, but you want to be able to ship with static setup for prod, usually. Good candidates * webservers * dynamically reloading WSGI servers * Programs watching a directory * etc --- # Building a Container, cont'd Now that we've demonstrated building a simple container, feel free to peruse the [Dockerfile reference](https://docs.docker.com/engine/reference/builder/) and .remark-inline-code[docker build --help] to get an idea of the rest of the capabilities of Dockerfiles. Utilizing other Dockerfile build directives like .remark-inline-code[RUN], .remark-inline-code[ENV], and .remark-inline-code[WORKDIR] it is possible to get a tremendous amount done. I highly recommend reading the entirety of the Dockerfile reference. ??? Fly! Be free! Build useful containers! You definitely want to read the docs, their is some interesting and powerful functionality which requires a bit of study - ENTRYPOINT, CMD - ARG, ENV - what can be done at run time vs build time --- # Your Code in a Container And now, employing the same basic formula as building your website container, combined with the knowledge of Dockerfile it is possible to Dockerize just about anything, including your own code! - Author the image - Select an appropriate base image - Perform the required setup, copying files or installing dependencies - Install your program - Specify a sane default command - Build the image - Run a container! ??? Write a Dockerfile Build Run Tweak --- # Your Code in a Container, cont'd A basic python example, assuming a module with a setup.py install process, might look as follows: Dockerfile ```dockerfile FROM python RUN apt-get install some_c_lib COPY . /code WORKDIR /code RUN pip install -r requirements.txt RUN python setup.py install CMD ['python' '-m' 'YourModuleName'] ``` and at the CLI ```bash $ docker build . -t mymodule $ docker run -d mymodule ``` ??? Python example is slightly more concrete Notice sometimes we have to jump through some hoops to install dependencies different ways --- # Additional Resources - .remark-inline-code[docker --help] - .remark-inline-code[docker run --help] - .remark-inline-code[docker build --help] - Really all the help documentation is great - [Dockerfile Reference](https://docs.docker.com/engine/reference/builder/) - [DockerHub](https://hub.docker.com/) - Video: [Jerome Petazzoni, AJ Bowen," Deploy and Scale Containers with Docker Native, Open Source Orchestration" - PyCon 2017](https://www.youtube.com/watch?v=EuzoEaE6Cqs) - Video: [Mike Coleman, "Docker?!? But I'm a SYSADMIN!" - Dockercon 2017](https://www.youtube.com/watch?v=M7ZBF-JJWVU) - The Dockercon youtube playlists - a bunch of great talks - If your language operandi has a convention I'd highly recommend looking up talks Docker may have given there, a lot of communities have been embracing Docker and doing cool language-specific stuff with it. - These slides are available online via [bnbalsamo.github.io](bnbalsamo.github.io) ??? The talks may be a bit lengthy, but are totally worth it Look up the Dockercon youtube playlists, watch anything that looks interesting Pycon also has some great talks, if you're into Python Don't be afraid to follow along --- class: middle .center[**Thank you**] .center[Questions?] .center[Brian Balsamo] .center[brian@brianbalsamo.com]