Add support for defining dnsPolicy and dnsConfig options for ECK operator statefulset

[haam3r]

This PR adds support to the statefulset template for defining dnsPolicy and dnsConfig options.

Flow control is designed to both prevent conflicts with the existing hostNetwork option and to only render out these options if they are both defined together.

If hostNetwork is enabled, dnsPolicy defaults to ClusterFirstWithHostNet unless explicitly set.

Both new options have been documented with examples in values.yaml.

[naemono]

Really sorry for the delays in getting this reviewed. I have a question about why we are making decisions on linking dnsPolicy and dnsConfig? And what if we set hostNetwork: true, and dnsPolicy: "something-else"? It explicitly sets dnsPolicy to clusterfirstWithHostNet.

[thbkrkr]

Thank you for this contribution! Since there hasn’t been recent activity, I’ll adopt this PR to complete it and get it merged. Your work will be preserved and credited.

[barkbay]

Should we add a link to the original documentation (https://kubernetes.io/docs/concepts/services-networking/dns-pod-service/#pod-s-dns-policy), and invite the reader to go there for more information?

[thbkrkr]

55073dd

[thbkrkr]

buildkite test this

[haam3r]

Really sorry for the delays in getting this reviewed. I have a question about why we are making decisions on linking dnsPolicy and dnsConfig? And what if we set hostNetwork: true, and dnsPolicy: "something-else"? It explicitly sets dnsPolicy to clusterfirstWithHostNet.

Sorry for the delay in responding, I missed the intial notification.
We are not specifically linking dnsPolicy and dnsConfig. They can still be defined separately, we are only statically setting dnsPolicy when hostnetwork is specified.
We explicitly set dnspolicy in that case because the k8s documentation says to do that. Otherwise as the docs say we introduce unintended behaviour. Per the docs linked in the code comment:

For Pods running with hostNetwork, you should explicitly set its DNS policy to "ClusterFirstWithHostNet". Otherwise, Pods running with hostNetwork and "ClusterFirst" will fallback to the behavior of the "Default" policy.