I have a dedicated server running Proxmox, which I use to provision some virtual machines for myself and friends. Recently I decided to move a bunch of Docker services I had running in a LXC container on the host to their own VM, mostly because I had some problems getting Tailscale working nicely on the container. While doing this, I also thought it might be a nice excuse to finally give Podman a try.
The initial move#
Installing Podman is pretty simple. I’m using Ubuntu 22.04 so it was already in the repos so an easy install with APT. It’s probably in your distro’s repositories as well.
sudo apt install podmanOne of the cool things about Podman is that it uses the standard fork/exec
technique seen around Unix (instead of a server/client model like Docker), and
the containers are run rootless by default (as the executing user instead). This
means that if you’re on a multi-user server, you don’t need to give users access
to sudo docker or to create a docker group for them, and it makes it more
easily auditable if you have the tools for that.
However because Podman is daemon-less, by default if you log out of your session
after starting some containers with it, you’ll find that they’ve mysteriously
disappeared. This is obviously annoying coming from Docker, but thankfully it’s
easily solvable with sudo loginctl enable-linger <username>, which then lets
them stay around once you log out.
Using Podman#
Moving to Podman from Docker turns out to be really easy. The team has put in a
lot of effort of making the CLI almost one-to-one with Docker, with the same
commands and flags. In fact, for the most part you can just alias docker to
podman, and carry on with your muscle memory from before.
$ podman run --rm docker.io/library/hello-world
Hello from Docker!
This message shows that your installation appears to be working correctly.
<snip>You’ll notice that I’m specifying the full name for the image I want to run,
instead of doing the shorthand hello-world. By default Podman doesn’t expand a
shorthand image to a fully qualified name, but there is a config option for
that. In /etc/containers/registries.conf (or
~/.config/containers/registries.conf) you can set the option
unqualified-search-registries to a list of domains to search through for a
matching image.
unqualified-search-registries = ["docker.io"]Besides this, Podman acts almost exactly how you would expect Docker would, which is really cool. Podman also provide their own alternative to Docker Compose called Podman Compose.
Podman Compose#
Podman Compose is easy to get, being in distro repos alongside Podman, and takes after Docker Compose v1 in the sense that it’s a Python CLI script which manages a group of containers and associated networks & volumes from a single manifest. For the most common usage you would want from Compose, Podman’s is a decent solution.
I personally have some gripes with some of Podman Compose’s behaviour though,
mainly in regards to how up works compared to Docker Compose. If you ever need
to update a compose file or the image one of the containers use, usually you
would just run docker-compose up and have Compose manage recreating the
related containers for you, however Podman Compose just doesn’t do this. Nothing
happens if you try to up an already running config, you need to take down the
entire deployment with docker-compose down and then run up again afterwards,
which is pretty annoying.
Additionally, the logs shown given by it are pretty painful. There’s a lot of noise where it has lines that look like something you would only expect to see with a debug option enabled. Call me a zoomer, but I much prefer how Docker Compose hides all that debug logging, and even gives you container labels + colour coding for each different log stream. It’s infinitely easier to understand what’s going on - especially if it’s not a service you’ve made yourself, where you can’t change the log format.
Thankfully, however, you can actually use Docker Compose natively with Podman without needing any weird hacks or doing anything stupid.
Docker Compose#
Yup, that’s right! Because of Podman wanting to make it as frictionless as
possible to migrate from Docker, they’re actually compatible enough on the
socket to natively support running Docker Compose. I’m going to be using Docker
Compose v2 in this because it has some niceties like docker-compose ls and
because it’s distributed as a single static Golang binary, making it easy to
download.
Now, Podman only added proper support for Compose v2 in v4.1.0, so let’s see if we have that.
$ podman -v
podman version 3.4.4
$ apt info podman
Package: podman
Version: 3.4.4+ds1-1ubuntu1Oh. That’s a bit of a problem.
(If you’re on a future version of Ubuntu (hopefully 23.04 has the right version), or on a different/rolling release distro, this following section probably won’t apply to you.)
Getting the correct Podman version#
Yup, it turns out Ubuntu 22.04 just missed out on getting the minimum version we
need in order to be able to use Docker Compose with it. Thankfully however,
since Ubuntu is based on Debian, we can actually yoink an updated .deb of the
version we want from their repos. Before that though, we need to install
libsubid4 for it. I tried with 4.13 from the Debian repos which didn’t work
(I forget the exact error I had), so instead I had to fetch 4.11.1 from
Ubuntu’s repos and use that, since that’s only available from >=22.10 as well.
wget https://security.ubuntu.com/ubuntu/pool/main/s/shadow/libsubid4_4.11.1+dfsg1-2ubuntu1.1_amd64.deb
sudo dpkg -i ibsubid4_4.11.1dfsg1-2ubuntu1.1_amd64_1.deb
wget http://ftp.au.debian.org/debian/pool/main/libp/libpod/podman_4.3.1+ds1-5+b1_amd64.deb
sudo dpkg -i podman_4.3.1+ds1-5+b1_amd64.deb$ podman -v
podman version 4.3.1Sweet!
NB: after doing this method, I encountered a bit of weirdness due to not having
containernetworking-plugins, some sort of packaging error I suppose, but this
was a simple fix by getting it via APT.
Getting Docker Compose#
So now we have a compatible version of Podman, we can turn our attention to getting Docker Compose. Ubuntu only ships 1.29 for some reason, so we need to v2 manually. Docker provides compiled executables on its GitHub releases page so we can just pull it from the latest release there.
wget https://github.com/docker/compose/releases/download/v2.15.0/docker-compose-linux-x86_64
sudo mv ./docker-compose-linux-x86_64 /usr/bin/docker-composeSimple.
Using it#
Before we can get started with Docker Compose, we first need to fool it into
thinking that Docker actually exists on our system. Usually it would expect a
Docker socket at unix:///var/run/docker.sock, but Podman obviously doesn’t
provide this, but if a $DOCKER_HOST variable is around, Compose will instead
prioritise using that. With this we can simply just pass our rootless Podman
socket as $DOCKER_HOST - by default this is at
unix://$XDG_RUNTIME_DIR/podman/podman.sock, if you’re on the first/sole user
on your system this will probably be unix:///run/user/1000/podman/podman.sock
like it was for me. Do double check though!
set -x DOCKER_HOST unix:///run/user/1000/podman/podman.sockEdit#
One thing I didn’t realise at the time of making this post originally, was that
this socket isn’t actually created by default. You will need to have Podman
launched as a service so that the socket exists, this can either be done with
the podman system service command or just by using systemd.
systemctl enable --user podman
systemctl start --user podmanAnd with this, we’re all set to use Docker Compose as we would if we were just
using normal Docker on our system, with re-running docker-compose up
restarting only the necessary containers as we would expect!
Keeping everything running with systemd#
Now we have everything running nicely with Podman & Docker Compose. However, there is one small problem we have right now. When we reboot the machine, our services don’t get brought up on startup. What??
This is due to Podman being a daemonless container runtime, so it has nothing that is around at startup time which can bring our containers online for us. Unless we intervene.
If we were using containers we had set up manually, and didn’t mind managing a
bunch of different unit files, Podman ships with podman generate systemd takes
a container name or ID and spits out a unit file along these lines:
# container-postgres.service
# autogenerated by Podman 4.3.1
# Sun Jan 8 10:45:24 UTC 2023
[Unit]
Description=Podman container-postgres.service
Documentation=man:podman-generate-systemd(1)
Wants=network-online.target
After=network-online.target
RequiresMountsFor=/run/user/1000/containers
[Service]
Environment=PODMAN_SYSTEMD_UNIT=%n
Restart=on-failure
TimeoutStopSec=60
ExecStart=/usr/bin/podman start postgres
ExecStop=/usr/bin/podman stop \
-t 0 postgres
ExecStopPost=/usr/bin/podman stop \
-t 0 postgres
PIDFile=/run/user/1000/containers/overlay-containers/46d92085dcde44ec376ab13e5a55ee4ad6b87e961934bd881e9b5cbaa204692d/userdata/conmon.pid
Type=forking
[Install]
WantedBy=default.targetHowever, if you’re using Compose configs, this can be a lot to manage depending on how many containers you’re managing in each one, and some services might need a specific start order/conditions to be met in order to start properly. And it’s also just a lot to do when you’re wanting to host something like Plausible or Sentry. So sadly this isn’t much use to us, but it is useful as a head start on creating our own units.
What I ended up doing was modifying it so that all the unit ends up doing is
starts the docker-compose up process with --detach so that it isn’t
controlling it outright. I do it this way instead of owning the compose process,
it exits after bringing anything up so that if I need to update a config or an
image, I can just do so with the compose CLI (it’s just what I prefer). However
if you have a service which exits at any time, by default systemd will consider
it finished andthus run the ExecStop command. Thankfully they provide two
options that counter this, making it consider such a service “alive” after it’s
finished running: Type=oneshot and RemainAfterExit=yes.
Knowing this, here is the unit file I came up with to run Docker Compose on boot:
[Unit]
Description=Databases compose config
Wants=network-online.target
After=network-online.target
RequiresMountsFor=/run/user/1000/containers
[Service]
Type=oneshot
RemainAfterExit=yes
Environment="PATH=/usr/local/bin:/usr/bin:/bin"
Environment="DOCKER_HOST=unix:///run/user/1000/podman/podman.sock"
ExecStart=docker-compose -f /home/ovy/dbs/docker-compose.yml up -d
ExecStop=docker-compose -f /home/ovy/dbs/docker-compose.yml down
[Install]
WantedBy=default.targetI’ll have this saved as dbs-compose.service.
Unfortunately you will still need to create a unit file for each compose config
you have (unless you create something like a shell script which does them for
you), but it’s still a lot better than if you had to do it for each container.
An additional note is that if you’re using $PWD in your compose configs, you
will need to rewrite them to use relative or absolute paths instead, as it will
instead point to some other location - probably the FS root.
To keep in line with the idea of Podman being rootless, I have these run as user
services under systemd, and it also just makes managing them easier in my
opinion. To do this you’ll need to put them into ~/.config/systemd/user/ and
then use systemctl --user to enable and start them running as your user
account.
mkdir -p ~/.config/systemd/user
# Copy/create all your unit files in that directory
systemctl --user daemon-reload
# Now to enable & start each service
systemctl --user enable db-compose.service
systemctl --user start db-compose.serviceNow if you run systemctl --user status my-compose.service you should see
something like this:
● dbs-compose.service - Databases compose config
Loaded: loaded (/home/ovy/.config/systemd/user/dbs-compose.service; enabled; v>
Active: active (exited) since Sun 2023-01-08 02:41:08 UTC; 22h ago
Process: 696 ExecStart=docker-compose -f /home/ovy/dbs/docker-compose.yml up -d>
Main PID: 696 (code=exited, status=0/SUCCESS)
CPU: 78ms
Jan 08 02:40:57 solai systemd[648]: Starting Databases compose config...
Jan 08 02:41:06 solai docker-compose[696]: Container minio Starting
Jan 08 02:41:06 solai docker-compose[696]: Container postgres Starting
Jan 08 02:41:07 solai docker-compose[696]: Container minio Started
Jan 08 02:41:08 solai docker-compose[696]: Container postgres Started
Jan 08 02:41:08 solai systemd[648]: Finished Databases compose config.Even though we can see the script has exited, it’s still being considered as
active thanks to our use of Type=oneshot and RemainAfterExit=yes. And now
when we reboot our machine, the services will come up automatically without us
needing to intervene!
If you have a compose setup where one config relies on containers another one
manages - for example, a single centralised database setup for other services -
you can utilise systemd’s
Wants/Requires
and
After
options to make sure they start in the correct order.
Overall, the move to Podman was relatively painless. Besides some packaging
unluckiness on Ubuntu’s part, it was easy to get up and running coming from
Docker, and the compatibility with existing tools like Docker Compose is a huge
plus too. It would be nice to see Podman Compose to be improved a bit to be up
to par with Docker Compose - at least in regards to re-up behaviour and logs -
as it is a good offering, but what’s there right now works.
This is the first blog post I’ve written ever, so let me know how it went, if there’s anything I can improve for future posts, etc. Thanks!