Using rootful podman-compose on Centos9

In the last weeks I updated my home server from Centos8 to Centos 9 stream. As part of this I also switched from docker to podman. This was no problem for single container but in case of multiple container working together - aka are composed - it wont work as expected. 

E.g in a setup of two container in a pod - app and db - the app can't resolve (and reach) the db. 

After multiple trial and error setups, crawling, reading through the web - filter out obsolete tips etc. I switch to "netavark" - (see https://github.com/containers/podman-compose/issues/455), be aware that the mentioned command podman system reset --force will remove all volumes (Yes, I did have a backup.)

In the end - all does not help ... I was about to give up. But then I found a hint how to look in the dns config of the container:

$ more /run/containers/networks/aardvark-dns/gogs_default
10.89.0.1
9951847ba473d97b6bf1e834b490 10.89.0.4 gogs_db,db,9951847ba473
6b06aee614d7584d32f3912e3b2c 10.89.0.5 gogs_app,app,6b06aee614d7

Ok, this mean the aardvark-dns is configured right. Next I look in the logs:

$ journalctl -t aardvark-dns
21:06:57 s...de aardvark-dns[1998]: Received SIGHUP will refresh servers: 1
21:06:57 s...de aardvark-dns[1998]: Unable to start server unable to start CoreDns server: \
                                    Address already in use (os error 98)

Looks like the aardvark-dns is in conflict with my name server runing on the same maschine.

$ ps 1998
 PID TIME COMMAND
1998 0:00 /usr/libexec/podman/aardvark-dns --config /run/containers/networks/aardvark-dns -p 53 run

Interesting - it is running on port 53 - like bind.

Finally I found out how to configure the port aardvark is using: it is a further entry in the file where I already switch to netavark.

$ more /etc/containers/containers.conf
[network]

# Explicitly use netavark. 
# See https://github.com/containers/podman-compose/issues/455
network_backend = "netavark"

dns_bind_port = 5533

Restart ... it runs.