We have a small selfhosted microk8s kubernetes cluster where CoreDNS (v1.6.6) provides the domain name resolution for the pods. A few days ago we run into the problem that some DNS request an application issues is resulting in NXDOMAIN errors. This only happens very rarely but causes the application to crash and restart because it cannot for example connect to the database as it cannot resolve the domain.

Our corefile configmap:

apiVersion: v1
data:
  Corefile: |
    .:53 {
        rewrite stop type AAAA A
        errors
        health {
            lameduck 5s
        }
        ready
        kubernetes cluster.local in-addr.arpa ip6.arpa {
            pods insecure
            fallthrough
            ttl 30
        }
        prometheus :9153
        forward . 1.1.1.1 8.8.8.8
        cache 30
        loop
        reload
        loadbalance
    }
kind: ConfigMap

We do need to rewrite AAAA to A as the provider does not support ipv6 currently.

One example of such an occurance: logfile of error

As you can see, the resolution of the database name works most of the time, but not always. Is there anything wrong with our Corefile that could cause this issue?