KeyVaultErrorException: Operation returned an invalid status code 'Forbidden'

Question

I'm trying to set up my web app, hosted in Azure to read settings from Azure KeyVault.

I've been following this guide: https://anthonychu.ca/post/secrets-aspnet-core-key-vault-msi/

The example shows how to access app settings from KeyVault with the configuration:

public static IWebHost BuildWebHost(string[] args) =>
    WebHost.CreateDefaultBuilder(args)
     .ConfigureAppConfiguration((ctx, builder) =>
     {
         var keyVaultEndpoint = Environment.GetEnvironmentVariable("KEYVAULT_ENDPOINT");
         if (!string.IsNullOrEmpty(keyVaultEndpoint))
         {
             var azureServiceTokenProvider = new AzureServiceTokenProvider();
             var keyVaultClient = new KeyVaultClient(
                 new KeyVaultClient.AuthenticationCallback(
                     azureServiceTokenProvider.KeyVaultTokenCallback));
             builder.AddAzureKeyVault(
                 keyVaultEndpoint, keyVaultClient, new DefaultKeyVaultSecretManager());
         }
     })
    .UseApplicationInsights()
    .UseStartup<Startup>()
    .Build();

I've added the KEYVAULT_ENDPOINT environment variable to the application settings. I've enabled MSI on the app service, and I've authorized my Azure User, and my application, from the Key Vault Access Policies:

With Get and List Operations:

And I've added the secret to key vault. Running locally, I can access the secret.

But my ASP .NET Core site fails on startup with this in the stdout logs:

Unhandled Exception: Microsoft.Azure.KeyVault.Models.KeyVaultErrorException: Operation returned an invalid status code 'Forbidden'
   at Microsoft.Azure.KeyVault.KeyVaultClient.GetSecretsWithHttpMessagesAsync(String vaultBaseUrl, Nullable`1 maxresults, Dictionary`2 customHeaders, CancellationToken cancellationToken)
   at Microsoft.Azure.KeyVault.KeyVaultClientExtensions.GetSecretsAsync(IKeyVaultClient operations, String vaultBaseUrl, Nullable`1 maxresults, CancellationToken cancellationToken)
   at Microsoft.Extensions.Configuration.AzureKeyVault.AzureKeyVaultConfigurationProvider.LoadAsync()
   at Microsoft.Extensions.Configuration.AzureKeyVault.AzureKeyVaultConfigurationProvider.Load()
   at Microsoft.Extensions.Configuration.ConfigurationRoot..ctor(IList`1 providers)
   at Microsoft.Extensions.Configuration.ConfigurationBuilder.Build()
   at Microsoft.AspNetCore.Hosting.WebHostBuilder.BuildCommonServices(AggregateException& hostingStartupErrors)
   at Microsoft.AspNetCore.Hosting.WebHostBuilder.Build()
   at Blog.Program.BuildWebHost(String[] args) in D:\a\1\s\[csproj name]\Program.cs:line 22
   at [csproj name].Program.Main(String[] args) in D:\a\1\s\[csproj name]\Program.cs:line 16

I've checked that MSI_ENDPOINT and MSI_SECRET environment variables exist by calling SET from the debug console.

I can also see the KEYVAULT_ENDPOINT variable.

Any suggestions what could be going wrong or on what to try next? Since it works locally it must be an authentication issue, but I believe it's authenticating locally with my azure User that I've authorized in key vault, rather than as the Azure App Service.

Solution 1

It is dodgy, but the problem with your setup is that you enabled "Authorized application" when you created the Access policy for your KeyValut.

You can see it because you have "Application+Application" on your screenshot. I imagine, you created the policy with your webapp being both principal and authrorized application. This kinda creates a loop.

To make this setup work just delete your existing policy and create a new one, where you select only the Principal:

Solution 2

only select principal and add web app,don't select authorized application

Solution 3

For me to resolve the issue, I had to add the IP addresses of the web app to the firewall as not all of them were getting added when you select "Allow trusted Microsoft services..."

Thanks to: https://azidentity.azurewebsites.net/post/2019/01/03/key-vault-firewall-access-by-azure-app-services

Solution 4

Check to ensure that the Service Principal you are using has been added and has the appropriate permissions for the operations you want to perform (Keys/Secrets etc):

Check to see if the KeyVault has restricted network access, toggle to test if this is the issue or not:

Solution 5

I want to add another case here with the same symptoms/error messages. Azure keyvault returned Forbidden when one of the secrets was disabled.

I had all the required permissions - I was accessing keyvault with Get and List granted for secrets (I was using only secrets) for the Managed Identity of my App Service.

It was confusing to get the "Access Forbidden" error, and it took some experimentation to figure out the issue. Hope this helps someone who have faced the same issue.

View more solutions