Unable to get access to Key Vault using Azure MSI on App Service

Question

I have enabled Managed Service Identities on an App Service. However, my WebJobs seem unable to access the keys.

They report:

Tried the following 3 methods to get an access token, but none of them worked. Parameters: Connectionstring: [No connection string specified], Resource: https://vault.azure.net, Authority: . Exception Message: Tried to get token using Managed Service Identity. Unable to connect to the Managed Service Identity (MSI) endpoint. Please check that you are running on an Azure resource that has MSI setup. Parameters: Connectionstring: [No connection string specified], Resource: https://vault.azure.net, Authority: https://login.microsoftonline.com/common. Exception Message: Tried to get token using Active Directory Integrated Authentication. Access token could not be acquired. password_required_for_managed_user: Password is required for managed user Parameters: Connectionstring: [No connection string specified], Resource: https://vault.azure.net, Authority: . Exception Message: Tried to get token using Azure CLI. Access token could not be acquired. 'az' is not recognized as an internal or external command,

Kudo does not show any MSI_ environmental variables.

How is this supposed to work? This is an existing App Service Plan.

Solution 1

The AppAuthentication library leverages an internal endpoint in App Service that receives the tokens on your site's behalf. This endpoint is non-static and therefore is set to an environment variable. After activating MSI for your site through ARM, your site will need to be restarted to get two new Environment Variables set in it:

MSI_ENDPOINT and MSI_SECRET

The presence of these variables are essential to the MSI feature working properly during runtime as the AppAuthentication library uses them to get the authorization token. The error message reflects this:

Exception Message: Tried to get token using Managed Service Identity. Unable to connect to the Managed Service Identity (MSI) endpoint. Please check that you are running on an Azure resource that has MSI setup.

If these variables are absent, you might need to restart the site.

https://docs.microsoft.com/en-us/azure/app-service/app-service-managed-service-identity

If the environment variables are set and you still see the same error, the article above has a code sample showing how to send requests to that endpoint manually.

public static async Task<HttpResponseMessage> GetToken(string resource, string apiversion)  {
HttpClient client = new HttpClient();
client.DefaultRequestHeaders.Add("Secret", Environment.GetEnvironmentVariable("MSI_SECRET"));
return await client.GetAsync(String.Format("{0}/?resource={1}&api-version={2}", Environment.GetEnvironmentVariable("MSI_ENDPOINT"), resource, apiversion));

}

I would try that and see what kind of response I get back.

Solution 2

I just solved this issue when trying to use MSI with a Function app, though I already had the environment variables set. I tried restarting multiple times to no success. What I ended up doing was manually turning off MSI for the Function, then re-enabling it. This wasn't ideal, but it worked.

Hope it helps!

Solution 3

I've found out that if you enable MSI and then swap out the slot, the functionality leaves with the slot change. You can re-enable it by switching it off and on again but that will create a new identity in AD and will require you to reset permissions on the key vault for it to work.

Solution 4

Enable the identity and give access to your azure function app in keyvault via access policy. You can find identity in platform feature tab These two steps works for me

Solution 5

For the folks that will come across these answers, I would like to share my experience.

I got this problem with Azure Synapse pipeline run. Essentially I added access policies properly to the KeyVault, and also I added a LinkedService to the Azure Synapse pointing to my KeyVault.

If I trigger the notebook manually it works, but in the pipeline, it fails.

Initially, I used the following statement:

url = TokenLibrary.getSecret("mykeyvault", "ConnectionString")

Then I added the name of the linked service as a third parameter, and the pipeline was able to leverage that linked service to obtain the MSI token for a Vault.

url = TokenLibrary.getSecret("mykeyvault", "ConnectionString", "AzureKeyVaultLinkedServiceName")

View more solutions

Comments

[Jerome Haltom]

So, I tried restarting the service. Numerous times. Nothing seems to make it work. Next step is to do it by hand. But, I feel that the problem is probably just that the env variable doesn't exist.

[Jerome Haltom]

I should add, everytime I deploy the site again (with an Arm template) the enable MSI is disabled. I have the arm template set up to SystemAssigned.

[James Christianson]

In your ARM template, make sure that the "Identity" block is not within the "Properties" block. You can go to resources.azure.com and look up your site to see it's ARM properties. If MSI is enabled, You should be able to find a TenantId and a PrincipalId within an "identity" object. In the portal, make sure the application setting WEBSITE_DISABLE_MSI is set to "false". If all of that fails, you should follow this article to allow the app service team to investigate into your case: github.com/projectkudu/kudu/wiki/…

[Jerome Haltom]

So, it looks like it's not surviving a slot swap. My deployment process is creating a new slot, deploying to it, then swapping with prod.

[James Christianson]

Ah I see. We are aware of an issue with MSI in deployment slots on app service, which might explain the issue you are hitting here and we are looking into getting it fixed.

[Jerome Haltom]

Well, I just turned off all that slot swap stuff. Now the policy stays applied, so that's a good start. But apps still can't load the stuff.

[Pelle]

Can anyone explain more about the significance of the WEBSITE_DISABLE_MSI app setting? What exactly does it do and when? I’m using terraform to set up the app service and MSI so I would need to this app setting in the tf file for terraform not to delete it.

[Sebastian Inones]

@JamesChristianson Is there any update on this? I'm experiencing the same issue: App Service with Deployment Slots and suddenly after a new code push the Prod slot it's unable to start, even though in staging it's working fine.

[Sebastian Inones]

I would be grateful then if you can vote up my answer.