The following guide is the second of a three part series to create a working ChatGPT alternative. The series will cover:
- Part 1: Setup and configuration of a Cloud Server using Hetzner.
- Part 2 (this guide): Setup and configuration of a
podmanenvironment to manage containers andtraefikto enable secure HTTPS access. - Installation and setup of both OpenWebUI and LiteLLM to manage and track API usage and a responsive front-end for LLMs.
The purpose of this guide is the installation and setup of podman to manage containers including traefik to provide secure web access over HyperText Transfer Protocol Secure (HTTPS). In addition, building from Part 1 how to provide additional security using the Hetzner firewall.
Linux Containers
Containers [1] (e.g. Docker) are lightweight and portable, sharing the host kernel to optimize resource efficiency. They encapsulate software and dependencies for consistent runtime across environments, excelling in CI/CD, web apps, and development.
MicroVMs [2] (e.g. Firecracker) provide stronger isolation by running applications in dedicated Virtual Machines (VMs), combining security with lightweight and fast boot times, ideal for high-security or untrusted workloads.
Hybrid approaches like Kata Containers [3] merge containers and microVMs, offering the portability and speed of containers within a VM for enhanced security, suitable for cloud-native apps requiring both agility and protection.
What is Docker?
When we think of Hoover or Dyson, we often associate those brand names with the general concept of vacuum cleaners. Similarly, Docker has become synonymous with containerization, encompassing a suite of tools and technologies that streamline application development and deployment.
Docker is an open-source platform that enables developers to package applications and their dependencies into standardized units called containers [1]. These containers can then run consistently across any environment.
Key Docker Components:
- Docker Daemon (
dockerd): This is the background service that manages Docker images, containers, networks, and storage volumes - Docker Client (
docker): This is the command-line interface (CLI) that users interact with to issue commands to the Docker daemon. - Docker Images: These are read-only templates that contain multiple layers of applications.
- Docker Containers: These are runnable instances of Docker images. They are isolated environments that contain everything needed to run an application using the hosts kernel.
- Docker Registry: This is a storage and distribution system for Docker images. Docker Hub is a public registry, while private registries can also be set up.
- Dockerfiles: These are text documents that contain all the commands a user could call on the command line to assemble an image.
- Docker Compose (
docker-compose): A tool for defining and running multi-container Docker applications.
- Docker Desktop: An application for Mac and Windows that provides an easy way to run Docker on user desktops.
- Docker Networks: These enable communication between Docker containers typically by modifying
iptablesand local routes on Linux hosts. - Docker Volumes: These provide persistent storage for Docker containers.
The list above is incomplete and doesn’t include tools such as BuildKit used to create container images.
What is Podman?
Podman [4] is a daemonless, open-source containerization tool designed to offer a more secure and efficient alternative to Docker. It allows users to run containers and pods [5] (as in Kubernetes [6] compatible multiple containers) without the need for a central daemon.
It’s key advantages over docker are:
- Daemonless Architecture: Unlike Docker, which relies on a central daemon,
podmanoperates without one, eliminating a single point of failure and enhancing security by reducing potential attack surfaces. - Rootless Containers:
podmansimplifies running containers without root (or admin) privileges. This improves security by minimizing the risk of system-wide impact from container vulnerabilities. - Compatibility with Docker:
podmanis compatible with Docker’s API and container images, allowing users to use existing Dockerfiles and commands with minimal adaptation. - Native
systemdIntegration:podmanintegrates smoothly withsystemdproviding a native, single host container orchestration.
Installing Podman
Log in to your Hetzner server and run the following commands to install podman:
# Update the package list
sudo apt update
[sudo] password for charlie:
Hit:1 https://mirror.hetzner.com/ubuntu/packages noble InRelease
Get:2 https://mirror.hetzner.com/ubuntu/packages noble-updates InRelease [126 kB]
Get:3 https://mirror.hetzner.com/ubuntu/packages noble-backports InRelease [126 kB]
Get:4 https://mirror.hetzner.com/ubuntu/security noble-security InRelease [126 kB]
Fetched 378 kB in 1s (562 kB/s)
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
All packages are up-to-date.
# Install podman
sudo apt install podman podman-compose
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
The following packages were automatically installed and are no longer required:
hwdata linux-tools-common
Use 'sudo apt autoremove' to remove them.
The following additional packages will be installed:
aardvark-dns buildah catatonit conmon containernetworking-plugins crun fuse-overlayfs golang-github-containers-common podman-compose python3-dotenv
golang-github-containers-image libgpgme11t64 libslirp0 libsubid4 libyajl2 netavark passt slirp4netns uidmap
Suggested packages:
containers-storage libwasmedge0 docker-compose
The following NEW packages will be installed
aardvark-dns buildah catatonit conmon containernetworking-plugins crun fuse-overlayfs golang-github-containers-common
golang-github-containers-image libgpgme11t64 libslirp0 libsubid4 libyajl2 netavark passt podman slirp4netns uidmap
0 to upgrade, 18 to newly install, 0 to remove and 0 not to upgrade.
Need to get 32.6 MB of archives.
After this operation, 132 MB of additional disk space will be used.
Do you want to continue? [Y/n]
<snip>
Scanning processes...
Scanning Linux images...
Running kernel seems to be up-to-date.
No services need to be restarted.
No containers need to be restarted.
No user sessions are running outdated binaries.
No VM guests are running outdated hypervisor (qemu) binaries on this host.
# Confirm the tool has been installed
podman version
Client: Podman Engine
Version: 4.9.3
API Version: 4.9.3
Go Version: go1.22.2
Built: Thu Jan 1 00:00:00 1970
OS/Arch: linux/amd64Unlike docker the podman tool requires full paths to pull images; for Docker Hub this is usually docker.io/library/<image>. As per the Tutorial we can use the nginx image to test our installation, note that we are running the command as a normal user without the use of sudo or other admin rights:
# Pull and run the nginx image as a container called basic_httpd
podman run --name basic_httpd -dt -p 8080:80/tcp docker.io/nginx
Trying to pull docker.io/library/nginx:latest...
Getting image source signatures
Copying blob 943ea0f0c2e4 done |
Copying blob 7cf63256a31a done |
Copying blob 513c3649bb14 done |
Copying blob bf9acace214a done |
Copying blob d014f92d532d done |
Copying blob 9dd21ad5a4a6 done |
Copying blob 103f50cb3e9f done |
Copying config b52e0b094b done |
Writing manifest to image destination
b1debf9897d38efc65b4be712b3467a275ae5e301b38c6bd120ab7ce98f1b291
# Check the status of the container
podman ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
b1debf9897d3 docker.io/library/nginx:latest nginx -g daemon o... 7 seconds ago Up 7 seconds 0.0.0.0:8080->80/tcp basic_httpdThe nginx container is now running and accessible on port 8080 of the server. To check the container is running correctly we can use curl to access the server. Note that Nginx is both a web server and a reverse proxy. The PORTS section shows the IP address as 0.0.0.0 which means it’s open to all networking interfaces on the server with a port of 8080. This is mapped internally in the container as port 80 which is the default port for the nginx web server.
Conduct two tests via two terminals. Run the first on the cloud server and use the IP address of 127.0.0.1, which is a way of saying localhost. The second test is from your local machine using the server’s public IP address:
Run the following command on the cloud server:
# Check access to the container using curl
curl 127.0.0.1:8080
<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
<style>
html { color-scheme: light dark; }
body { width: 35em; margin: 0 auto;
font-family: Tahoma, Verdana, Arial, sans-serif; }
</style>
</head>
<body>
<h1>Welcome to nginx!</h1>
<p>If you see this page, the nginx web server is successfully installed and
working. Further configuration is required.</p>
<p>For online documentation and support please refer to
<a href="http://nginx.org/">nginx.org</a>.<br/>
Commercial support is available at
<a href="http://nginx.com/">nginx.com</a>.</p>
<p><em>Thank you for using nginx.</em></p>
</body>
</html>Run the following command on your local machine:
# Check access to the container using curl
curl 159.77.77.207:8080
<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
<style>
html { color-scheme: light dark; }
body { width: 35em; margin: 0 auto;
font-family: Tahoma, Verdana, Arial, sans-serif; }
</style>
</head>
<body>
<h1>Welcome to nginx!</h1>
<p>If you see this page, the nginx web server is successfully installed and
working. Further configuration is required.</p>
<p>For online documentation and support please refer to
<a href="http://nginx.org/">nginx.org</a>.<br/>
Commercial support is available at
<a href="http://nginx.com/">nginx.com</a>.</p>
<p><em>Thank you for using nginx.</em></p>
</body>
</html>Note both outputs are the same, this is because the nginx container is running on the cloud server and accessible from both the server and your local machine. The next step is to install traefik to provide secure access over HTTPS.
Traefik
Introduction
Traefik [7] is an open-source, modern reverse-proxy, and load balancing tool designed to simplify the management of containerized applications. This makes it effective routing traffic to the appropriate application running typically in containers and balancing the load across all available nodes.
It dynamically routes incoming HyperText Transport Protocol (HTTP) requests to appropriate services, eliminating the need for manual port management. Traefik integrates seamlessly with container orchestration tools like Kubernetes, Docker Swarm and Podman, automatically discovering containers and updating its configurations accordingly. This uses the native features of these orchestration tools such as labels. This means the user can set the routing configuration directly in the container orchestration declarative file such as a docker-compose.yml or a pod definition.
It also supports automation of Let’s Encrypt [8] SSL/TLS certificates, ensuring secure communication. Essentially, Traefik acts as a bridge between the internet and containerized applications, enhancing security, efficiency, and ease of use in managing application traffic.
The guide is based on the excellent blog by Chris Hudson [9] but adapted to be more file based using the docker-compose format.
Prerequisites
Enable the socket for podman to provide API [10] access to traefik:
# Start the socket
systemctl --user start podman.socket
# Auto start the socker by enabling it
systemctl --user enable podman.socket
Created symlink /home/charlie/.config/systemd/user/sockets.target.wants/podman.socket → /usr/lib/systemd/user/graphical-session-pre.target.wants/
# Check its status
systemctl --user status podman.socket
● podman.socket - Podman API Socket
Loaded: loaded (/usr/lib/systemd/user/podman.socket; enabled; preset: enabled)
Active: active (listening) since Mon 2025-03-10 21:29:17 UTC; 31s ago
Triggers: ● podman.service
Docs: man:podman-system-service(1)
Listen: /run/user/1000/podman/podman.sock (Stream)
CGroup: /user.slice/user-1000.slice/[email protected]/app.slice/podman.socket
Mar 10 21:29:17 chatty-cheetah systemd[5736]: Listening on podman.socket - Podman AP>Note next to the key Loaded it states enabled and next to the key Active it states active demonstrating the socket is working.
Host Setup
Port Access
Typically for HTTPS traffic the default port is 443, however on Linux insallations ports below 1024 are reserved for the root user. To avoid running traefik as root we need to provide unprivileged users to bind privileged ports. This can be done by modifying the Linux kernel using the sysctl tool as a root user or the sudo command:
# Set the start of unprivileged ports to 443 for this session
sudo sysctl net.ipv4.ip_unprivileged_port_start=443
[sudo] password for charlie:
net.ipv4.ip_unprivileged_port_start = 443
# Set the same setting permanently
echo "net.ipv4.ip_unprivileged_port_start=443" | sudo tee /etc/sysctl.d/user_priv_ports.conf
net.ipv4.ip_unprivileged_port_start=443
# Verify changes for current session
sudo sysctl net.ipv4.ip_unprivileged_port_start
net.ipv4.ip_unprivileged_port_start = 443
# Verify changes for permanent settings
sudo cat /proc/sys/net/ipv4/ip_unprivileged_port_start
443SSL/TLS Certificates
Create a directory to store the settings for engaging with Lets Encrypt and creating an empty file for the acme.json file:
TODO: Lets Encrypt service definition.
# Create directory
mkdir traefik && cd traefik
# Create acme file
touch acme.json
# Change permissions
chmod 600 acme.jsonDomain Name Service (DNS) Records
The Domain Name Service (DNS) converts a domain like example.com into an IP address using nameservers. These are for configured typically by your domain provider unless you have opted for another service or party to manage them on your behalf. For example, it’s common when using third-part Content Delivery Networks (CDNs) like Cloudflare [11] or Akamai [12] to manage the DNS records for you.
The DNS records for the domain name must be set to the public IP address of the server. The traefik service will then map the domain to the container. Each domain provider has its own method (and help guide) but if using an IPv4 address then an A record is required; for IPv6 an AAAA record is required.
Confirm DNS records have been set using the dig tool or a public DNS lookup service as those from Google’s Admin Toolbox [13]:
# Check latest DNS records
dig dandy.rootinsights.io
; <<>> DiG 9.18.30-0ubuntu0.24.04.2-Ubuntu <<>> dandy.rootinsights.io
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 21030
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1
;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 65494
;; QUESTION SECTION:
;dandy.rootinsights.io. IN A
;; ANSWER SECTION:
dandy.rootinsights.io. 300 IN A 159.77.77.207
;; Query time: 25 msec
;; SERVER: 127.0.0.53#53(127.0.0.53) (UDP)
;; WHEN: Mon Mar 10 22:10:30 UTC 2025
;; MSG SIZE rcvd: 66Configure and Launch Traefik
Use the following docker-compose file:
version: '3'
services:
traefik:
image: docker.io/library/traefik:latest
container_name: traefik
restart: unless-stopped
networks:
- podman
security_opt:
- label=type:container_runtime_t
volumes:
1 - /run/user/<user_id>/podman/podman.sock:/var/run/docker.sock:z
2 - ./acme.json:/acme.json:z
ports:
- "443:443"
command:
- --api.dashboard=true
- --api.insecure=true
3 - --certificatesresolvers.lets-encrypt.acme.email=<email_address>
- --certificatesresolvers.lets-encrypt.acme.storage=/acme.json
- --certificatesresolvers.lets-encrypt.acme.tlschallenge=true
- --entrypoints.https.address=:443
- --providers.docker=true
labels:
# Dashboard router
- "traefik.enable=true"
4 - "traefik.http.routers.dashboard.rule=Host(`dandy.rootinsights.io`)"
- "traefik.http.routers.dashboard.entrypoints=https"
- "traefik.http.routers.dashboard.service=api@internal"
- "traefik.http.routers.dashboard.tls=true"
- "traefik.http.routers.dashboard.tls.certresolver=lets-encrypt"
networks:
podman:
external: true- 1
-
Enter the user id (obtained by running
id <username>) in the path. - 2
- Map the internal file name for the Lets Encrypt service (left hand side) to the container file name.
- 3
- Enter a valid email address for the Lets Encrypt service.
- 4
- Specify the domain name for the dashboard.
There key variable needed to ensure other containers can utilise traefik effectively is the entrypoints variable, in this instance called https. If this is changed to say websecure then it must be propergated through all the container definitions.
In order for the domain to map to the container, the Domain Name System (DNS) record must be set to the public IP address of the server. The traefik service will then map the domain to the container. Each domain provider has its own method (and help guide) but if using an IPv4 address then an A record is required; for IPv6 an AAAA record is required.
Run the file as an unprivileged user (note no use of the sudo command):
# Run and test compose file
podman compose up
>>>> Executing external compose provider "/usr/bin/podman-compose". Please refer to the documentation for details. <<<<
podman-compose version: 1.0.6
['podman', '--version', '']
using podman version: 4.9.3
** excluding: set()
['podman', 'ps', '--filter', 'label=io.podman.compose.project=traefik', '-a', '--format', '{{ index .Labels "io.podman.compose.config-hash"}}']
recreating: ...
** excluding: set()
podman stop -t 10 traefik
traefik
exit code: 0
podman rm traefik
traefik
exit code: 0
recreating: done
['podman', 'network', 'exists', 'podman']
podman create --name=traefik --security-opt label=type:container_runtime_t --label traefik.enable=true --label traefik.http.routers.dashboard.rule=Host(`dandy.rootinsights.io`) --label traefik.http.routers.dashboard.entrypoints=https --label traefik.http.routers.dashboard.service=api@internal --label traefik.http.routers.dashboard.tls=true --label traefik.http.routers.dashboard.tls.certresolver=lets-encrypt --label io.podman.compose.config-hash=904074c0fd852603ab5c1af81afb3ae06b336bdadc172722f58f574ea229cc37 --label io.podman.compose.project=traefik --label io.podman.compose.version=1.0.6 --label [email protected] --label com.docker.compose.project=traefik --label com.docker.compose.project.working_dir=/home/charlie/traefik --label com.docker.compose.project.config_files=docker-compose.yml --label com.docker.compose.container-number=1 --label com.docker.compose.service=traefik -v /run/user/1000/podman/podman.sock:/var/run/docker.sock:z -v /home/charlie/traefik/acme.json:/acme.json:z --net podman --network-alias traefik -p 443:443 --restart unless-stopped docker.io/library/traefik:latest --api.dashboard=true --api.insecure=true --certificatesresolvers.lets-encrypt.acme.email=<email_address> --certificatesresolvers.lets-encrypt.acme.storage=/acme.json --certificatesresolvers.lets-encrypt.acme.tlschallenge=true --entrypoints.https.address=:443 --providers.docker=true
4afe8390a3f8da692167e229889810347354e3a19b16d964f466515cf650dd45
exit code: 0
podman start -a traefik
[traefik] | 2025-03-10T22:19:24Z ERR error="service \"basic-httpd\" error: unable to
find the IP address for the container \"/basic_httpd\": the server is ignored" container=basic-httpd-b1debf9897d38efc65b4be712b3467a275ae5e301b38c6bd120ab7ce98f1b291 providerName=dockerIn another terminal check the status of the traefik container:
# List running containers
podman ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
d67e4ad68c7d docker.io/library/traefik:latest --api.dashboard=t... 31 seconds ago Up 30 seconds 0.0.0.0:443->443/tcp traefikWe can now access the dashabord via the domain name we specified earlier:
Press Ctrl-CCtrl-C to stop the container running in the foreground and remove the existing container as we intend to use the same name:
# Kill current container
^CTraceback (most recent call last):
File "/usr/bin/podman-compose", line 33, in <module>
sys.exit(load_entry_point('podman-compose==1.0.6', 'console_scripts', 'podman-compose')())
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3/dist-packages/podman_compose.py", line 2941, in main
podman_compose.run()
File "/usr/lib/python3/dist-packages/podman_compose.py", line 1423, in run
cmd(self, args)
File "/usr/lib/python3/dist-packages/podman_compose.py", line 1754, in wrapped
return func(*args, **kw)
^^^^^^^^^^^^^^^^^
File "/usr/lib/python3/dist-packages/podman_compose.py", line 2117, in compose_up
thread.join(timeout=1.0)
File "/usr/lib/python3.12/threading.py", line 1151, in join
self._wait_for_tstate_lock(timeout=max(timeout, 0))
File "/usr/lib/python3.12/threading.py", line 1167, in _wait_for_tstate_lock
if lock.acquire(block, timeout):
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
KeyboardInterrupt
Error: unable to start container d67e4ad68c7d0bb0203419c1cb99ae9fe295d5805f57bd5fd25bdcda17d5f234: attaching to container d67e4ad68c7d0bb0203419c1cb99ae9fe295d5805f57bd5fd25bdcda17d5f234: write /dev/stdout: broken pipe
# Check if container is running
podman ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
d67e4ad68c7d docker.io/library/traefik:latest --api.dashboard=t... 9 minutes ago Up About a minute 0.0.0.0:443->443/tcp traefik
# Stop the container
podman stop traefik
traefik
# List previously run containers
podman ps -a
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS
NAMES
b1debf9897d3 docker.io/library/nginx:latest nginx -g daemon o... 5 days ago Exited (0) 5 days ago 0.0.0.0:8080->80/tcp
basic_httpd
d67e4ad68c7d docker.io/library/traefik:latest --api.dashboard=t... 8 minutes ago Up 45 seconds 0.0.0.0:443->443/tcp
traefik
# Remove container `traefik`
podman container rm traefik
traefikWe can now rerun the compose file and the containers within in the background as a process daemon with the -d flag:
# Run the compose file in the background
podman compose up -d
>>>> Executing external compose provider "/usr/bin/podman-compose". Please refer to the documentation for details. <<<<
podman-compose version: 1.0.6
['podman', '--version', '']
using podman version: 4.9.3
** excluding: set()
['podman', 'ps', '--filter', 'label=io.podman.compose.project=traefik', '-a', '--format', '{{ index .Labels "io.podman.compose.config-hash"}}']
['podman', 'network', 'exists', 'podman']
podman run --name=traefik -d --security-opt label=type:container_runtime_t --label traefik.enable=true --label traefik.http.routers.dashboard.rule=Host(`dandy.rootinsights.io`) --label traefik.http.routers.dashboard.entrypoints=https --label traefik.http.routers.dashboard.service=api@internal --label traefik.http.routers.dashboard.tls=true --label traefik.http.routers.dashboard.tls.certresolver=lets-encrypt --label io.podman.compose.config-hash=904074c0fd852603ab5c1af81afb3ae06b336bdadc172722f58f574ea229cc37 --label io.podman.compose.project=traefik --label io.podman.compose.version=1.0.6 --label [email protected] --label com.docker.compose.project=traefik --label com.docker.compose.project.working_dir=/home/charlie/traefik --label com.docker.compose.project.config_files=docker-compose.yml --label com.docker.compose.container-number=1 --label com.docker.compose.service=traefik -v /run/user/1000/podman/podman.sock:/var/run/docker.sock:z -v /home/charlie/traefik/acme.json:/acme.json:z --net podman --network-alias traefik -p 443:443 --restart unless-stopped docker.io/library/traefik:latest --api.dashboard=true --api.insecure=true --certificatesresolvers.lets-encrypt.acme.email=<email_address> --certificatesresolvers.lets-encrypt.acme.storage=/acme.json --certificatesresolvers.lets-encrypt.acme.tlschallenge=true --entrypoints.https.address=:443 --providers.docker=true
b3bf470d13b549d88489649e6e2eead0952cb3b5987fa962ab990971baad62d4
exit code: 0Test Automatic DNS Provisioning
The purpose of traefik is to automatically provision Transport Layer Security (TLS) certificates for a given domain name, secure its traffic with HTTPS and route the traffic to the correct container, just using the labels feature of a docker-compose file and therefore without any complex configuration files. We can test this with another container, typically used to confirm container management software (an there are many to choose from) called crcchek/hello-word:
# Create a new folder
mkdir ~/hello && cd ~/hello
# Copy the previous docker-compose file or create a new one
cp ../traefik/docker-compose.yml .
# Or
touch docker-compose.ymlEdit the file for the following content:
version: '3'
services:
hello:
image: docker.io/crccheck/hello-world
container_name: hello
hostname: hello.rootinsights.io
labels:
- traefik.enable=true
- traefik.http.routers.hello-secure.entrypoints=https
- traefik.http.routers.hello-secure.rule=Host(`hello.rootinsights.io`)
- traefik.http.middlewares.hello-https-redirect.redirectscheme.scheme=https
- traefik.http.routers.hello.middlewares=hello-https-redirect
- traefik.http.routers.hello-secure.tls=true
- traefik.http.routers.hello-secure.tls.certresolver=lets-encrypt
- traefik.http.services.hello.loadbalancer.server.port=8000Ensure that you have created the DNS records for the domain name to point to the public IP address of the server i.e. hello.domain.com points to the IP address of your public server (use ip -o -4 addr show dev eth0 | awk '{print $4}' | cut -d'/' -f1).
We can test using the curl command or visiting the address directly in a web browser:
curl https://hello.rootinsights.io
<pre>
Hello World
## .
## ## ## ==
## ## ## ## ## ===
/""""""""""""""""\___/ ===
~~~ {~~ ~~~~ ~~~ ~~~~ ~~ ~ / ===- ~~~
\______ o _,/
\ \ _,'
`'--.._\..--''
</pre>We can use the dashboard to better understand the routing and security features of traefik:
The image above reiterates the name of each entry point, in this case port 443 is labelled https. We can click on the Services area to determine what containers are connected to the traefik service:
We can further explore a given service, in this case the hello service:
Any error messages can be found in this section, it also shows the address and port of the container itself. The other as aspect to explore is the routers section:
Note the two green shields to indicate secure through TLS and the nature of the provider (i.e. Docker) even though we are using podman. We can select the external hello-secure@docker router to get more details:
The diagram at the top summarises where this router sits in that it’s between the Entrypoint and the Service. Having illustrated that additional containers can be started and connected to traefik for efficient routing and secure access using HTTPS we can either disable the hello container or re-run it as a daemon.
Persistence
Whilst the host is up, the containers should keep running but what happens if there is an error and the service crashes or the host loses power or is turned off? Currently, the containers require manual intervention to restart. To automate this process we can use the systemd service manager to create a service file to start the traefik service on boot and the other containers as required.
There a number of pre-requisites to create a systemd service file:
# Check if the systemd user instance is running
systemctl --user status
● chatty-cheetah
State: running
Units: 153 loaded (incl. loaded aliases)
Jobs: 0 queued
Failed: 0 units
Since: Sun 2025-03-16 10:10:29 UTC; 1h 41min ago
systemd: 255.4-1ubuntu8.5
CGroup: /user.slice/user-1000.slice/[email protected]
├─app.slice
│ └─podman.service
│ └─21406 /usr/bin/podman --log-level=info system service
├─init.scope
│ ├─20405 /usr/lib/systemd/systemd --user
│ └─20406 "(sd-pam)"
├─session.slice
│ └─dbus.service
│ └─20593 /usr/bin/dbus-daemon --session --address=systemd: --nofork --nopidfile --systemd-activation --syslog-only
└─user.slice
├─libpod-b3bf470d13b549d88489649e6e2eead0952cb3b5987fa962ab990971baad62d4.scope
│ └─container
│ └─21393 traefik traefik --api.dashboard=true --api.insecure=true --certificatesresolvers.lets-encrypt.acme.emai>
├─libpod-conmon-b3bf470d13b549d88489649e6e2eead0952cb3b5987fa962ab990971baad62d4.scope
│ └─21391 /usr/bin/conmon --api-version 1 -c b3bf470d13b549d88489649e6e2eead0952cb3b5987fa962ab990971baad62d4 -u b3>
├─podman-21316.scope
│ ├─21378 rootlessport
│ └─21383 rootlessport-child
├─podman-pause-7e7badf0.scope
│ └─20581 catatonit -P
└─rootless-netns-72e6d007.scope
└─21328 /usr/bin/slirp4netns --disable-host-loopback --mtu=65520 --enable-sandbox --enable-seccomp --enable-ipv6 >
# Check podman is running rootless i.e. as unprivileged user
podman infoThe are some guides on how to use the podman tool with systemd most notably from its authors, Red Hat [14] but the documentation is out of date. The current approach is use a declarative file format to generate the systemd unit files using a technology called a quadlet introduced by Dan Walsh in a blog post for podman 4.4 [15].
Let’s start with the traeik container. Usin the now deprecated podman generate systemd command we can create a systemd unit file:
# Generate a systemd unit file for the traefik container
podman generate systemd --name traefik --new --restart-policy=always > ~/.config/systemd/user/traefik.service
DEPRECATED command:
It is recommended to use Quadlets for running containers and pods under systemd.
Please refer to podman-systemd.unit(5) for details.
# Check contents of systemd file
cat ~/.config/systemd/user/traefik.service
# container-traefik.service
# autogenerated by Podman 4.9.3
# Sun Mar 16 14:31:03 UTC 2025
[Unit]
Description=Podman container-traefik.service
Documentation=man:podman-generate-systemd(1)
Wants=network-online.target
After=network-online.target
RequiresMountsFor=%t/containers
[Service]
Environment=PODMAN_SYSTEMD_UNIT=%n
Restart=always
TimeoutStopSec=70
ExecStart=/usr/bin/podman run \
--cidfile=%t/%n.ctr-id \
--cgroups=no-conmon \
--rm \
--sdnotify=conmon \
--replace \
--name=traefik \
-d \
--security-opt label=type:container_runtime_t \
--label traefik.enable=true \
--label traefik.http.routers.dashboard.rule=Host(`dandy.rootinsights.io`) \
--label traefik.http.routers.dashboard.entrypoints=https \
--label traefik.http.routers.dashboard.service=api@internal \
--label traefik.http.routers.dashboard.tls=true \
--label traefik.http.routers.dashboard.tls.certresolver=lets-encrypt \
--label io.podman.compose.config-hash=904074c0fd852603ab5c1af81afb3ae06b336bdadc172722f58f574ea229cc37 \
--label io.podman.compose.project=traefik \
--label io.podman.compose.version=1.0.6 \
--label [email protected] \
--label com.docker.compose.project=traefik \
--label com.docker.compose.project.working_dir=/home/charlie/traefik \
--label com.docker.compose.project.config_files=docker-compose.yml \
--label com.docker.compose.container-number=1 \
--label com.docker.compose.service=traefik \
-v /run/user/1000/podman/podman.sock:/var/run/docker.sock:z \
-v /home/charlie/traefik/acme.json:/acme.json:z \
--net podman \
--network-alias traefik \
-p 443:443 docker.io/library/traefik:latest \
--api.dashboard=true \
--api.insecure=true \
--certificatesresolvers.lets-encrypt.acme.email=<email_address> \
--certificatesresolvers.lets-encrypt.acme.storage=/acme.json \
--certificatesresolvers.lets-encrypt.acme.tlschallenge=true \
--entrypoints.https.address=:443 \
--providers.docker=true
ExecStop=/usr/bin/podman stop \
--ignore -t 10 \
--cidfile=%t/%n.ctr-id
ExecStopPost=/usr/bin/podman rm \
-f \
--ignore -t 10 \
--cidfile=%t/%n.ctr-id
Type=notify
NotifyAccess=all
[Install]
WantedBy=default.targetThe flag -new is intended to create the container if it doesn’t already exist and most importantly the flag --name is for identifying an existing container to manage. Check the status of the new service:
# Check the status of the traefik service
systemctl --user status traefik.service
○ traefik.service - Podman container-traefik.service
Loaded: loaded (/home/charlie/.config/systemd/user/traefik.service; disabled; preset: enabled)
Active: inactive (dead)
Docs: man:podman-generate-systemd(1)
# Start it
systemctl --user start traefik.service
# Check status
systemctl --user status traefik.service
● traefik.service - Podman container-traefik.service
Loaded: loaded (/home/charlie/.config/systemd/user/traefik.service; disabled; preset: enabled)
Active: active (running) since Sun 2025-03-16 14:39:04 UTC; 2s ago
Docs: man:podman-generate-systemd(1)
Main PID: 25371 (conmon)
Tasks: 14 (limit: 4540)
Memory: 4.8M (peak: 16.8M)
CPU: 455ms
CGroup: /user.slice/user-1000.slice/[email protected]/app.slice/traefik.service
├─25358 rootlessport
├─25363 rootlessport-child
└─25371 /usr/bin/conmon --api-version 1 -c dc115a837dee0f0e4fd7ded2d41ed51a05efcb69213296a1563bf37f537596f9 -u dc11>
Mar 16 14:39:04 chatty-cheetah systemd[23709]: Starting traefik.service - Podman container-traefik.service...
Mar 16 14:39:04 chatty-cheetah podman[25295]: 2025-03-16 14:39:04.445604615 +0000 UTC m=+0.084052690 container remove b3bf470d13>
Mar 16 14:39:04 chatty-cheetah podman[25295]: 2025-03-16 14:39:04.467599556 +0000 UTC m=+0.106047587 container create dc115a837d>
Mar 16 14:39:04 chatty-cheetah podman[25295]: 2025-03-16 14:39:04.415456823 +0000 UTC m=+0.053904849 image pull e49b83f3e049cba3>
Mar 16 14:39:04 chatty-cheetah podman[25295]: 2025-03-16 14:39:04.761482864 +0000 UTC m=+0.399930940 container init dc115a837dee>
Mar 16 14:39:04 chatty-cheetah podman[25295]: 2025-03-16 14:39:04.774941862 +0000 UTC m=+0.413389937 container start dc115a837de>
Mar 16 14:39:04 chatty-cheetah systemd[23709]: Started traefik.service - Podman container-traefik.service.
Mar 16 14:39:04 chatty-cheetah podman[25295]: dc115a837dee0f0e4fd7ded2d41ed51a05efcb69213296a1563bf37f537596f9
# Enable at boot
systemctl --user enable traefik.service
Created symlink /home/charlie/.config/systemd/user/default.target.wants/traefik.service → /home/charlie/.config/systemd/user/traefik.service.We can test by stopping the traefik container manually:
# Stop the container
podman stop traefik
traefik
# Check status (note it has restarted)
podman ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
0323b47330c0 docker.io/library/traefik:latest --api.dashboard=t... 31 seconds ago Up 31 seconds 0.0.0.0:443->443/tcp traefik
# Check the journal logs
journalctl --user -u traefik.service
Mar 16 14:39:04 chatty-cheetah systemd[23709]: Starting traefik.service - Podman container-traefik.service...
Mar 16 14:39:04 chatty-cheetah podman[25295]: 2025-03-16 14:39:04.445604615 +0000 UTC m=+0.084052690 container remove b3bf470d13>
Mar 16 14:39:04 chatty-cheetah podman[25295]: 2025-03-16 14:39:04.467599556 +0000 UTC m=+0.106047587 container create dc115a837d>
Mar 16 14:39:04 chatty-cheetah podman[25295]: 2025-03-16 14:39:04.415456823 +0000 UTC m=+0.053904849 image pull e49b83f3e049cba3>
Mar 16 14:39:04 chatty-cheetah podman[25295]: 2025-03-16 14:39:04.761482864 +0000 UTC m=+0.399930940 container init dc115a837dee>
Mar 16 14:39:04 chatty-cheetah podman[25295]: 2025-03-16 14:39:04.774941862 +0000 UTC m=+0.413389937 container start dc115a837de>
Mar 16 14:39:04 chatty-cheetah systemd[23709]: Started traefik.service - Podman container-traefik.service.
Mar 16 14:39:04 chatty-cheetah podman[25295]: dc115a837dee0f0e4fd7ded2d41ed51a05efcb69213296a1563bf37f537596f9
Mar 16 14:44:54 chatty-cheetah traefik[25371]: 2025-03-16T14:44:54Z ERR error="accept tcp [::]:8080: use of closed network conne>
Mar 16 14:44:54 chatty-cheetah traefik[25371]: 2025-03-16T14:44:54Z ERR error="close tcp [::]:8080: use of closed network connec>
Mar 16 14:44:54 chatty-cheetah traefik[25371]: 2025-03-16T14:44:54Z ERR error="accept tcp [::]:443: use of closed network connec>
Mar 16 14:44:54 chatty-cheetah traefik[25371]: 2025-03-16T14:44:54Z ERR error="close tcp [::]:443: use of closed network connect>
Mar 16 14:44:55 chatty-cheetah podman[25752]: 2025-03-16 14:44:55.625054815 +0000 UTC m=+0.297059413 container remove dc115a837d>
Mar 16 14:44:55 chatty-cheetah systemd[23709]: traefik.service: Scheduled restart job, restart counter is at 1.Notice the last line of the log that states Scheduled restart job... For a more realistic test, we can start the hello container previously with the -d flag and then restart the host. Note that traefik is supported by the systemd, whereas the hello container has no orchestrator defined:
# Navigate to the hello directory
cd ~/hello
# Either remove existing container, amend compose file with replace flag and then start container
podman compose up -d
>>>> Executing external compose provider "/usr/bin/podman-compose". Please refer to the documentation for details. <<<<
podman-compose version: 1.0.6
['podman', '--version', '']
using podman version: 4.9.3
** excluding: set()
['podman', 'ps', '--filter', 'label=io.podman.compose.project=hello', '-a', '--format', '{{ index .Labels "io.podman.compose.config-hash"}}']
recreating: ...
** excluding: set()
podman stop -t 10 hello
hello
exit code: 0
podman rm hello
hello
exit code: 0
recreating: done
['podman', 'network', 'exists', 'hello_default']
podman run --name=hello -d --label traefik.enable=true --label traefik.http.routers.hello-secure.entrypoints=https --label traefik.http.routers.hello-secure.rule=Host(`hello.rootinsights.io`) --label traefik.http.middlewares.hello-https-redirect.redirectscheme.scheme=https --label traefik.http.routers.hello.middlewares=hello-https-redirect --label traefik.http.routers.hello-secure.tls=true --label traefik.http.routers.hello-secure.tls.certresolver=lets-encrypt --label traefik.http.services.hello.loadbalancer.server.port=8000 --label io.podman.compose.config-hash=6091c48699dc500027d62df90c0bed36986f668e84d4d6ab2a39d1b5c61b2eff --label io.podman.compose.project=hello --label io.podman.compose.version=1.0.6 --label [email protected] --label com.docker.compose.project=hello --label com.docker.compose.project.working_dir=/home/charlie/hello --label com.docker.compose.project.config_files=docker-compose.yml --label com.docker.compose.container-number=1 --label com.docker.compose.service=hello --net hello_default --network-alias hello --hostname hello.rootinsights.io docker.io/crccheck/hello-world
5b46ef67c505a644073f9ebf46eac3e492d3a68319700803736fddf2b8a3bbeb
exit code: 0
# Check container status
podman ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS
NAMES
0323b47330c0 docker.io/library/traefik:latest --api.dashboard=t... 8 minutes ago Up 8 minutes 0.0.0.0:443->443/tcp traefik
5b46ef67c505 docker.io/crccheck/hello-world:latest /bin/sh -c echo "... 7 seconds ago Up 8 seconds (healthy)
hello
# Reboot the host
sudo reboot
# Post reboot check status
podman ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
405cb202d1b3 docker.io/library/traefik:latest --api.dashboard=t... 3 seconds ago Up 3 seconds 0.0.0.0:443->443/tcp traefikNote that only the traefik container has been restarted; this demonstrates that the systemd unit is working as intended.
Securing Access
We saw from the earlier example with nginx that it’s possible to open a port to the public internet, which can have unintended consequences. Unfortunately, most container engines and associated management tools bypass local (as in on the host) firewall rules meaning tools such as iptables, nftables or ufw are ineffective [16].
Hetnzer provides a firewall service that can be used to secure access to the server. The service is available in the Cloud Console under the Network section. The firewall rules are applied to the public interface of the server and can be used to restrict access to specific ports and IP addresses.
The aim is to restrict inbound traffic to the SSH port defined in Part 1 of the guide and port 443 for HTTPS traffic. The outbound traffic is restricted to non-private IP addresses. A common attack process is to takeover a host and conduct a scan of machines local to the host. These will have IP ranges that fall into the IPv4 Private Address Space [17], typically in the range:
10.0.0.0/8172.16.0.0/12192.168.0.0/16
Log into the console to access the dashboard:
Click on Firewalls:
Set the first rule to allow SSH traffic, followed by responding to ping request via the ICMP protocol and finally HTTPS traffic via port 443:
Restrict the outbound traffic to exclude the Private Internet Address Space:
Apply the firewall and waiting for it to become active:
Whilst this securely locks down all other ports, naturally traffic on port 443 is open and in particular the traefik dashboard is open to the wider internet. Whilst its possible to provide local access control, a more effective means is to use a third party CDN such as Cloudflare to provide a Web Application Firewall (WAF) and DDoS protection. In particular the use of Cloudflare Zero Trust:
Conclusion
This guide showed how to use podman to manage containers and in particular the podman-compose tool to orchestrate the using a a declarative file format. We used traefik as a reverse proxy to route traffic to the correct container and secure it with HTTPS. We then used systemd to manage the traefik container and other containers to ensure they started automatically after a reboot of the host and attempted to recover due to a crash or abnormality. Finally, we secured access to the server using the Hetzner firewall service.
Attribution
Images based on:
- Image by chiến nguyễn bá from Pixabay
References
Citation
@online{2025,
author = {, miah0x41},
title = {Managing {Containers} and {Providing} {Secure} {Access} to
{You} {Own} {*ChatGPT*} {Interface}},
date = {2025-03-22},
url = {https://blog.curiodata.pro/posts/13-podman-traefik/},
langid = {en}
}