How to find out IOPS disk usage by pod/container on k8s nodes

Why is there a need of defining one's deployment's labels multiple times?

Because they are for different audiences, and they do different things. That's usually the case for when you encounter something in computers that appears to be a duplicate

I'll skip to the tl;dr before answering the rest of your questions: if your deployments use the same values for all 3 labels, then you can use yaml anchors to stop the copy-paste:

apiVersion: apps/v1
kind: Deployment
metadata:
  labels: &labels
    app: app
    tier: backend
spec:
  selector:
    matchLabels: *labels
  template:
    metadata:
      labels: *labels

My first issue is that I really couldn't find any good reason for the spec.selector.matchlabels to exist at all. spec.selector.matchlabels must match spec.template.metadata.labels right?

The selector:matchLabels: must be a subset of the labels applied to the Pod, but they don't need to match exactly, otherwise it would limit the labels that could be applied to a Pod, since additional ones would then invalidate the matchLabels: and the Pod would no longer be managed by the Deployment.

My second issue, which might have a more reasonable answer, is why do you have to explicitely define deployment's labels and template's labels? Aren't they supposed to match each other as well or are there any cases where they might not match to each other? Am I just using them wrong? There's no clear documentation over their use against template's labels. :/

The Deployment labels are metadata for your own use -- the kubernetes scheduler does not consult them in the slightest, to the very best of my knowledge. Pods are the unit of scheduling: everything else is just accounting for the kubernetes controller-manager to keep track of the desired state of the world. If you wanted to be able to quickly identify all Deployments that were created by the "secops" team, you could say kubectl get deploy -l team=secops and it would cheerfully give you back the list. Or release=alpha, or deployed-by=gitlab or whatever. It's just metadata.

The template metadata is how you define labels that will be applied to objects which do not yet exist -- the Deployment is by definition describing a future state of the world, and thus you need to describe how you want things to be labeled when they are created. Then, you need to be able to describe the set of all Pods which are considered "members" of the Deployment. It's both possible, and I've even used it a few times, to create a new Deployment with a selector: which matches existing Pods -- and in that case, the Deployment will take over responsibility for those Pods and so long as they match the described criteria, nothing in the Pod scheduling will change, but they are, in fact, now owned by a Deployment and it will supervise them.

After trying a lot of options , I tried to debug my deployment configuration file that I was using to deploy the pod to AKS cluster. I had a host mount based volume that was pointing to the path "/var/run".

Once I removed the host mount , the curl and wget worked as expected.

After discussing this behaviour with MS support, they confirmed that curl and wget are not falling back to /etc/resolv.conf file for DNS resolution if you have a host mount pointed to path "/var/run" may be due to the way DNS probing is implemented in curl and wget.