Disable the firewall if you can to check if that’s the issue, then do a tcpdump using root with the port. Do tcpdump inside the container too and compare what you see to the docker environment.

Is caddy-caddy really the correct image?

Try with this command, it’s the minimal setup that works by default (on my machine): podman run -p 0.0.0.0:5050:80 docker.io/library/caddy:latest

Use ss -tlpn or podman ps to show what ports podman is listening on, my guess is it is only listening on localhost.

Either use ipv6 privacy extension (enabled by default, so this can just be called ipv6) or don’t enable ipv6.

That way you have working ipv6 or wait until you come to your senses. Using nat6 ipv6 isn’t worth it.