Azure – Assign Network Contributor Role to AKS Cluster via ARM/Bicep

aksazureazure-arm-templateazure-networkingkubernetes

I'm trying to configure a Load Balancer for my AKS server using Bicep/ARM. I am using the NGinx Ingress Controller in kubernetes and it does seem to work but when I first spin things up I am encountering an error.

Mainly I'm wondering what is the equivalent ARM or Bicep template for this step in the Azure documentation?

https://docs.microsoft.com/en-us/azure/aks/static-ip#create-a-service-using-the-static-ip-address

az role assignment create \
    --assignee <Client ID> \
    --role "Network Contributor" \
    --scope /subscriptions/<subscription id>/resourceGroups/<resource group name>

I'm using Bicep and have created my AKS server like this for example:

resource ExampleKubernetes 'Microsoft.ContainerService/managedClusters@2021-07-01' = {
  // ...
}

I'm then adding a role assignment to the kubelet identity like so:

var NetworkContibutor = '4d97b98b-1d4f-4787-a291-c67834d212e7'
resource AssignNetworkContributorToKubelet 'Microsoft.Authorization/roleAssignments@2020-08-01-preview' = {
  name: guid(resourceGroup().id, ExampleKubernetes.id, NetworkContibutor)
  dependsOn: [
    ExampleKubernetes
  ]
  properties: {
    roleDefinitionId: subscriptionResourceId('Microsoft.Authorization/roleDefinitions', NetworkContibutor)
    principalType: 'ServicePrincipal'
    principalId: ExampleKubernetes.properties.identityProfile.kubeletidentity.objectId
  }
}

Which seems to work, I can see the Role assigned to the managed principal in the dashboard… but the Service in kubernetes seems to fail with a permission issue still:

  Error syncing load balancer: failed to ensure load balancer: Retriable: false,
  RetryAfter: 0s, HTTPStatusCode: 403, RawError: Retriable: false, RetryAfter:
  0s, HTTPStatusCode: 403, RawError:
  {"error":{"code":"AuthorizationFailed","message":"The client
  '<some guid A>' with object id
  '<some buid A>' does not have authorization to perform
  action 'Microsoft.Network/publicIPAddresses/read' over scope
  '/subscriptions/<subid>/resourceGroups/example/providers/Microsoft.Network/publicIPAddresses/example'
  or the scope is invalid. If access was recently granted, please refresh your
  credentials."}}

What's weird is that later on at some point it seems to just magically work. That error says "retriable false" and it does seem like the service doesn't retry but a subsequent deploy of NGinx to kubernetes will then cause it to retry and suddenly boom its working.

It just seems like the error message is telling me there is some non-deterministic delay of role propagation… So my questions are:

  • Is that right? Is it in fact just a delay and my code is basically right?
  • Am I using the right principalId? Or is that actually unnecessary?
  • Is there a way for me to force those role updates to propagate? I could have a CLI step in between if I needed to. How can I wait to install my ingress controller which connects to the LB after the permissions are ready?

Best Answer

I am not sure why previous answer considered correct. You have used kubelet identity here. This is used to authenticate with azure container registries, but in your case you must use Cluster (Control Plane) Identity and I can't find a way how to assign a System Managed Identity. I believe the only way at the moment is to bring your own.

1 Add Managed Identity creation to your ARM template:

resource managedidentity 'Microsoft.ManagedIdentity/userAssignedIdentities@2023-01-31' = {
  name: aksManIdentityName
  location: location
}

2 Update AKS Identity property:

  identity:{
    type: 'UserAssigned'
    userAssignedIdentities: {
      '${managedidentity.id}': {}
    }
  }

3 Grant Network Contributor Permissions

resource RBAC_Network_Contributor 'Microsoft.Authorization/roleAssignments@2022-04-01' = {
  scope: resourceGroup()
  name: guid(resourceGroup().id, '${aksClusterName}-NetworkContributor') 
  properties: {
    roleDefinitionId: resourceId('Microsoft.Authorization/roleDefinitions', '4d97b98b-1d4f-4787-a291-c67834d212e7')
    principalId: managedidentity.properties.principalId
    principalType: 'ServicePrincipal'
  }
}
Related Topic