Summary
I have an Azure App Service running a custom container. When I bind a path to an Azure Files share my container stops working. Looking at the Container Issues
logs, I see the error: [BYOS] Custom storage volume(s) failed to initialize: [/var/LWASFiles/Sites/my-app/a3484543-39f9-45a3-816b-9524640dfd50]
.
Detail
-
My custom container has a volume defined
/var/www/html/v3/uploads
. -
I'm attempting to map this to an Azure Files share, which is on a storage account in the same vnet as the hosting ASE, with rules permitting traffic between the ASE's subnet and the file share on ports 137, 138, 138, and 445.
-
The hosting ASE is configured with an internal load balancer (i.e. it's a private ASE).
-
The storage account has a private endpoint configured.
-
The volume is mapped under the App Service's Settings > Configuration > Path mappings, using a valid key and a mount path which matches the volume path on the container.
-
Before adding the path mapping, the app service runs as expected.
-
After adding the path mapping, the container fails to start / the only exception information I can find is from the
Container Issues
logs, as per the summary above. -
If I remove the path mapping, the issue persists, even after forcing a restart. The only way to resolve the issue is to drop and redeploy the app service.
-
It successfully connects to a MySql DB (Azure Database for MySQL servers) via a private endpoint.
-
Using
dig {privateEndpointFqdn}
I'm able to prove that the Storage Account's private endpoint is resolved correctly (as is the MySQL's; as you'd expect) -
Using
tcptraceroute {privateEndpointFqdn} {destinationPort}
I'm able to prove that I can connect to the storage account on port 445 (and to the MySQL DB on port 3306). -
Note: I can't connect to the storage account on ports 137, 138, or 139 despite those being allowed via the NSG on the same in & outbound rules as are used for 445, above; though I suspect those ports are no longer required for CIFS (I only added these as a just-in-case measure after first hitting the issue in case they were somehow relevant as they were mentioned in some posts).
-
I'm running the above tests by SSHing into the container, so the commands are run from its context. Note: Because I can't start the container after adding the path mapping, these tests are performed after creating the app service but before adding the path mapping.
-
I've increased
WEBSITES_CONTAINER_START_TIME_LIMIT
to its max value:1800
-
My container exposes port 80 (i.e. one of the default ports supported by App Service for container); and the website works when the path mapping doesn't exist, so this shouldn't be the issue. I've also set
WEBSITES_PORT
to80
, just for belt-and-braces. -
WEBSITES_ENABLE_APP_SERVICE_STORAGE
is set tofalse
(though I've also experimented with setting it totrue
, just in case / it made no difference so I reverted tofalse
). -
My file share has several GBs of data. I've tried creating an identical file share with no content and mapping to that (immediately after recreating the app service and proving that that worked; to ensure my tests weren't impacted by the hangover of the larger file share). This gives the same issue as with the larger file share.
-
The image for my container is being loaded from an Azure Container Repo.
-
The image is based on the
ubuntu:21.04
-
I've included the
azure-cli
andcifs-utils
packages (I think this is only required when performing the mount operation from the container, rather than from the AppService's config; but I wanted to cover all assumptions) -
The hosting ASE is in the
UK South
region (as is the storage account / all resources).
Note: This is my first experience with running containers in App Service; so PEBKAC is definitely a possibility.
Any suggestions or troubleshooting advise would be much appreciated. Thank-you.
Best Answer
I resolved the issue by changing the storage account's network from
selected networks
(in which I'd whitelisted by ASE's VNet and the outbound IPs for the web app) toall networks
via the portal:https://portal.azure.com/#@<myTentantName>.onmicrosoft.com/resource/subscriptions/<mySubscrption>/resourceGroups/<myStorageAccountsResourceGroup>/providers/Microsoft.Storage/storageAccounts/<myStorageAccountName>/networking
I then restarted my app (including dropping and re-adding the mount... not sure if that was required, but for good measure):