In the last weeks I updated my home server from Centos8 to Centos 9 stream. As part of this I also switched from docker to podman. This was no problem for single container but in case of multiple container working together - aka are composed - it wont work as expected.
E.g in a setup of two container in a pod - app and db - the app can't resolve (and reach) the db.
After multiple trial and error setups, crawling, reading through the web - filter out obsolete tips etc. I switch to "netavark" - (see https://github.com/containers/podman-compose/issues/455), be aware that the mentioned command podman system reset --force
will remove all volumes (Yes, I did have a backup.)
In the end - all does not help ... I was about to give up. But then I found a hint how to look in the dns config of the container:
$ more /run/containers/networks/aardvark-dns/gogs_default 10.89.0.1 9951847ba473d97b6bf1e834b490 10.89.0.4 gogs_db,db,9951847ba473 6b06aee614d7584d32f3912e3b2c 10.89.0.5 gogs_app,app,6b06aee614d7
Ok, this mean the aardvark-dns is configured right. Next I look in the logs:
$ journalctl -t aardvark-dns 21:06:57 s...de aardvark-dns[1998]: Received SIGHUP will refresh servers: 1 21:06:57 s...de aardvark-dns[1998]: Unable to start server unable to start CoreDns server: \ Address already in use (os error 98)Looks like the aardvark-dns is in conflict with my name server runing on the same maschine.
$ ps 1998 PID TIME COMMAND 1998 0:00 /usr/libexec/podman/aardvark-dns --config /run/containers/networks/aardvark-dns -p 53 runInteresting - it is running on port 53 - like bind.
Finally I found out how to configure the port aardvark is using: it is a further entry in the file where I already switch to netavark.
$ more /etc/containers/containers.conf [network] # Explicitly use netavark. # See https://github.com/containers/podman-compose/issues/455 network_backend = "netavark" dns_bind_port = 5533Restart ... it runs.