Encrypted configuration in ASP.NET Core

Question

With web.config going away, what is the preferred way to store sensitive info (passwords, tokens) in the configurations of a web app built using ASP.NET Core?

Is there a way to automatically get encrypted configuration sections in appsettings.json?

Solution 1

User secrets looks like a good solution for storing passwords, and, generally, application secrets, at least during development.

Check the official Microsoft documentation. You can also review this other SO question.

This is just a way to "hide" your secrets during development process and to avoid disclosing them into the source tree; the Secret Manager tool does not encrypt the stored secrets and should not be treated as a trusted store.

If you want to bring an encrypted appsettings.json to production, you can do so by building a custom configuration provider.

For example:

public class CustomConfigProvider : ConfigurationProvider, IConfigurationSource
{
    public CustomConfigProvider()
    {
    }

    public override void Load()
    {
        Data = UnencryptMyConfiguration();
    }

    private IDictionary<string, string> UnencryptMyConfiguration()
    {
        // do whatever you need to do here, for example load the file and unencrypt key by key
        //Like:
       var configValues = new Dictionary<string, string>
       {
            {"key1", "unencryptedValue1"},
            {"key2", "unencryptedValue2"}
       };
       return configValues;
    }

    private IDictionary<string, string> CreateAndSaveDefaultValues(IDictionary<string, string> defaultDictionary)
    {
        var configValues = new Dictionary<string, string>
        {
            {"key1", "encryptedValue1"},
            {"key2", "encryptedValue2"}
        };
        return configValues;                
    }

    public IConfigurationProvider Build(IConfigurationBuilder builder)
    {
       return new CustomConfigProvider();
    }
}

Define a static class for your extension method:

public static class CustomConfigProviderExtensions
{              
        public static IConfigurationBuilder AddEncryptedProvider(this IConfigurationBuilder builder)
        {
            return builder.Add(new CustomConfigProvider());
        }
}

And then you can activate it:

// Set up configuration sources.
var builder = new ConfigurationBuilder()
    .AddJsonFile("appsettings.json")
    .AddEncryptedProvider()
    .AddJsonFile($"appsettings.{env.EnvironmentName}.json", optional: true);

Solution 2

I agree with @CoderSteve that writing a whole new provider is too much work. It also doesn't build on the existing standard JSON architecture. Here is a solution that I come up with the builds on top of the standard JSON architecture, uses the preferred .Net Core encryption libraries, and is very DI friendly.

public static class IServiceCollectionExtensions
{
    public static IServiceCollection AddProtectedConfiguration(this IServiceCollection services)
    {
        services
            .AddDataProtection()
            .PersistKeysToFileSystem(new DirectoryInfo(@"c:\keys"))
            .ProtectKeysWithDpapi();

        return services;
    }

    public static IServiceCollection ConfigureProtected<TOptions>(this IServiceCollection services, IConfigurationSection section) where TOptions: class, new()
    {
        return services.AddSingleton(provider =>
        {
            var dataProtectionProvider = provider.GetRequiredService<IDataProtectionProvider>();
            section = new ProtectedConfigurationSection(dataProtectionProvider, section);

            var options = section.Get<TOptions>();
            return Options.Create(options);
        });
    }

    private class ProtectedConfigurationSection : IConfigurationSection
    {
        private readonly IDataProtectionProvider _dataProtectionProvider;
        private readonly IConfigurationSection _section;
        private readonly Lazy<IDataProtector> _protector;

        public ProtectedConfigurationSection(
            IDataProtectionProvider dataProtectionProvider,
            IConfigurationSection section)
        {
            _dataProtectionProvider = dataProtectionProvider;
            _section = section;

            _protector = new Lazy<IDataProtector>(() => dataProtectionProvider.CreateProtector(section.Path));
        }

        public IConfigurationSection GetSection(string key)
        {
            return new ProtectedConfigurationSection(_dataProtectionProvider, _section.GetSection(key));
        }

        public IEnumerable<IConfigurationSection> GetChildren()
        {
            return _section.GetChildren()
                .Select(x => new ProtectedConfigurationSection(_dataProtectionProvider, x));
        }

        public IChangeToken GetReloadToken()
        {
            return _section.GetReloadToken();
        }

        public string this[string key]
        {
            get => GetProtectedValue(_section[key]);
            set => _section[key] = _protector.Value.Protect(value);
        }

        public string Key => _section.Key;
        public string Path => _section.Path;

        public string Value
        {
            get => GetProtectedValue(_section.Value);
            set => _section.Value = _protector.Value.Protect(value);
        }

        private string GetProtectedValue(string value)
        {
            if (value == null)
                return null;

            return _protector.Value.Unprotect(value);
        }
    }
}

Wire up your protected config sections like this:

public void ConfigureServices(IServiceCollection services)
{
    services.AddMvc();

    // Configure normal config settings
    services.Configure<MySettings>(Configuration.GetSection("MySettings"));

    // Configure protected config settings
    services.AddProtectedConfiguration();
    services.ConfigureProtected<MyProtectedSettings>(Configuration.GetSection("MyProtectedSettings"));
}

You can easily create encrypted values for your config files using a controller like this:

[Route("encrypt"), HttpGet, HttpPost]
public string Encrypt(string section, string value)
{
    var protector = _dataProtectionProvider.CreateProtector(section);
    return protector.Protect(value);
}

Usage: http://localhost/cryptography/encrypt?section=SectionName:KeyName&value=PlainTextValue

Solution 3

I didn't want to write a custom provider – way too much work. I just wanted to tap into JsonConfigurationProvider, so I figured out a way that works for me, hope it helps someone.

public class JsonConfigurationProvider2 : JsonConfigurationProvider
{
    public JsonConfigurationProvider2(JsonConfigurationSource2 source) : base(source)
    {
    }

    public override void Load(Stream stream)
    {
        // Let the base class do the heavy lifting.
        base.Load(stream);

        // Do decryption here, you can tap into the Data property like so:

         Data["abc:password"] = MyEncryptionLibrary.Decrypt(Data["abc:password"]);

        // But you have to make your own MyEncryptionLibrary, not included here
    }
}

public class JsonConfigurationSource2 : JsonConfigurationSource
{
    public override IConfigurationProvider Build(IConfigurationBuilder builder)
    {
        EnsureDefaults(builder);
        return new JsonConfigurationProvider2(this);
    }
}

public static class JsonConfigurationExtensions2
{
    public static IConfigurationBuilder AddJsonFile2(this IConfigurationBuilder builder, string path, bool optional,
        bool reloadOnChange)
    {
        if (builder == null)
        {
            throw new ArgumentNullException(nameof(builder));
        }
        if (string.IsNullOrEmpty(path))
        {
            throw new ArgumentException("File path must be a non-empty string.");
        }

        var source = new JsonConfigurationSource2
        {
            FileProvider = null,
            Path = path,
            Optional = optional,
            ReloadOnChange = reloadOnChange
        };

        source.ResolveFileProvider();
        builder.Add(source);
        return builder;
    }
}

Solution 4

I managed to create a custom JSON configuration provider which uses DPAPI to encrypt and decrypt secrets. It basically uses simple regular expressions that you can define to specify what parts of the JSON needs to be encrypted.

The following steps are performed:

1. Json file is loaded
2. Determine whether the JSON parts that match the given regular expressions are already encrypted (or not). This is done by base-64 decoding of the JSON part and verify whether it starts with the expected prefix !ENC!)
3. If not encrypted, then encrypt the JSON part by first using DPAPI and secondly add the prefix !ENC! and encode to base-64
4. Overwrite the unencrypted JSON parts with the encrypted (base-64) values in the Json file

Note that the base-64 does not bring better security, but only hides the prefix !ENC! for cosmetic reasons. This is just a matter of taste of course ;)

This solution consists of the following classes:

1. ProtectedJsonConfigurationProvider class (= custom JsonConfigurationProvider)
2. ProtectedJsonConfigurationSource class (= custom JsonConfigurationSource)
3. AddProtectedJsonFile() extension method on the IConfigurationBuilder in order to simple add the protected configuration

Assuming the following initial authentication.json file:

{
    "authentication": {
        "credentials": [
            {
                user: "john",
                password: "just a password"
            },
            {
                user: "jane",
                password: "just a password"
            }
        ]
    }
}

Which becomes (sort of) the following after loading

{
    "authentication": {
        "credentials": [
            {
                "user": "john",
                "password": "IUVOQyEBAAAA0Iyd3wEV0R=="
            },
            {
                "user": "jane",
                "password": "IUVOQyEBAAAA0Iyd3wEV0R=="
            }
        ]
    }
}

And assuming the following configuration class based on the json format

public class AuthenticationConfiguration
{
    [JsonProperty("credentials")]
    public Collection<CredentialConfiguration> Credentials { get; set; }
}

public class CredentialConfiguration
{
    [JsonProperty("user")]
    public string User { get; set; }
    [JsonProperty("password")]
    public string Password { get; set; }
}

Below the sample code:

//Note that the regular expression will cause the authentication.credentials.password path to be encrypted.
//Also note that the byte[] contains the entropy to increase security
var configurationBuilder = new ConfigurationBuilder()
    .AddProtectedJsonFile("authentication.json", true, new byte[] { 9, 4, 5, 6, 2, 8, 1 },
        new Regex("authentication:credentials:[0-9]*:password"));

var configuration = configurationBuilder.Build();
var authenticationConfiguration = configuration.GetSection("authentication").Get<AuthenticationConfiguration>();

//Get the decrypted password from the encrypted JSON file.
//Note that the ProtectedJsonConfigurationProvider.TryGet() method is called (I didn't expect that :D!)
var password = authenticationConfiguration.Credentials.First().Password

Install the Microsoft.Extensions.Configuration.Binder package in order to get the configuration.GetSection("authentication").Get<T>() implementation

And finally the classes in which the magic happens :)

/// <summary>Represents a <see cref="ProtectedJsonConfigurationProvider"/> source</summary>
public class ProtectedJsonConfigurationSource : JsonConfigurationSource
{
    /// <summary>Gets the byte array to increse protection</summary>
    internal byte[] Entropy { get; private set; }

    /// <summary>Represents a <see cref="ProtectedJsonConfigurationProvider"/> source</summary>
    /// <param name="entropy">Byte array to increase protection</param>
    /// <exception cref="ArgumentNullException"/>
    public ProtectedJsonConfigurationSource(byte[] entropy)
    {
        this.Entropy = entropy ?? throw new ArgumentNullException(Localization.EntropyNotSpecifiedError);
    }

    /// <summary>Builds the configuration provider</summary>
    /// <param name="builder">Builder to build in</param>
    /// <returns>Returns the configuration provider</returns>
    public override IConfigurationProvider Build(IConfigurationBuilder builder)
    {
        EnsureDefaults(builder);
        return new ProtectedJsonConfigurationProvider(this);
    }

    /// <summary>Gets or sets the protection scope of the configuration provider. Default value is <see cref="DataProtectionScope.CurrentUser"/></summary>
    public DataProtectionScope Scope { get; set; }
    /// <summary>Gets or sets the regular expressions that must match the keys to encrypt</summary>
    public IEnumerable<Regex> EncryptedKeyExpressions { get; set; }
}

/// <summary>Represents a provider that protects a JSON configuration file</summary>
public partial class ProtectedJsonConfigurationProvider : JsonConfigurationProvider
{
    private readonly ProtectedJsonConfigurationSource protectedSource;
    private readonly HashSet<string> encryptedKeys = new HashSet<string>(StringComparer.OrdinalIgnoreCase);
    private static readonly byte[] encryptedPrefixBytes = Encoding.UTF8.GetBytes("!ENC!");

    /// <summary>Checks whether the given text is encrypted</summary>
    /// <param name="text">Text to check</param>
    /// <returns>Returns true in case the text is encrypted</returns>
    private bool isEncrypted(string text)
    {
        if (text == null) { return false; }

        //Decode the data in order to verify whether the decoded data starts with the expected prefix
        byte[] decodedBytes;
        try { decodedBytes = Convert.FromBase64String(text); }
        catch (FormatException) { return false; }

        return decodedBytes.Length >= encryptedPrefixBytes.Length
            && decodedBytes.AsSpan(0, encryptedPrefixBytes.Length).SequenceEqual(encryptedPrefixBytes);
    }

    /// <summary>Converts the given key to the JSON token path equivalent</summary>
    /// <param name="key">Key to convert</param>
    /// <returns>Returns the JSON token path equivalent</returns>
    private string convertToTokenPath(string key)
    {
        var jsonStringBuilder = new StringBuilder();

        //Split the key by ':'
        var keyParts = key.Split(':');
        for (var keyPartIndex = 0; keyPartIndex < keyParts.Length; keyPartIndex++)
        {
            var keyPart = keyParts[keyPartIndex];

            if (keyPart.All(char.IsDigit)) { jsonStringBuilder.Append('[').Append(keyPart).Append(']'); }
            else if (keyPartIndex > 0) { jsonStringBuilder.Append('.').Append(keyPart); }
            else { jsonStringBuilder.Append(keyPart); }
        }

        return jsonStringBuilder.ToString();
    }

    /// <summary>Writes the given encrypted key/values to the JSON oconfiguration file</summary>
    /// <param name="encryptedKeyValues">Encrypted key/values to write</param>
    private void writeValues(IDictionary<string, string> encryptedKeyValues)
    {
        try
        {
            if (encryptedKeyValues == null || encryptedKeyValues.Count == 0) { return; }

            using (var stream = new FileStream(this.protectedSource.Path, FileMode.Open, FileAccess.ReadWrite))
            {
                JObject json;

                using (var streamReader = new StreamReader(stream, Encoding.UTF8, true, 4096, true))
                {
                    using (var jsonTextReader = new JsonTextReader(streamReader))
                    {
                        json = JObject.Load(jsonTextReader);

                        foreach (var encryptedKeyValue in encryptedKeyValues)
                        {
                            var tokenPath = this.convertToTokenPath(encryptedKeyValue.Key);
                            var value = json.SelectToken(tokenPath) as JValue;
                            if (value.Value != null) { value.Value = encryptedKeyValue.Value; }
                        }
                    }
                }

                stream.Seek(0, SeekOrigin.Begin);
                using (var streamWriter = new StreamWriter(stream))
                {
                    using (var jsonTextWriter = new JsonTextWriter(streamWriter) { Formatting = Formatting.Indented })
                    {
                        json.WriteTo(jsonTextWriter);
                    }
                }
            }
        }
        catch (Exception exception)
        {
            throw new Exception(string.Format(Localization.ProtectedJsonConfigurationWriteEncryptedValues, this.protectedSource.Path), exception);
        }
    }

    /// <summary>Represents a provider that protects a JSON configuration file</summary>
    /// <param name="source">Settings of the source</param>
    /// <see cref="ArgumentNullException"/>
    public ProtectedJsonConfigurationProvider(ProtectedJsonConfigurationSource source) : base(source)
    {
        this.protectedSource = source as ProtectedJsonConfigurationSource;
    }

    /// <summary>Loads the JSON data from the given <see cref="Stream"/></summary>
    /// <param name="stream"><see cref="Stream"/> to load</param>
    public override void Load(Stream stream)
    {
        //Call the base method first to ensure the data to be available
        base.Load(stream);

        var expressions = protectedSource.EncryptedKeyExpressions;
        if (expressions != null)
        {
            //Dictionary that contains the keys (and their encrypted value) that must be written to the JSON file
            var encryptedKeyValuesToWrite = new Dictionary<string, string>();

            //Iterate through the data in order to verify whether the keys that require to be encrypted, as indeed encrypted.
            //Copy the keys to a new string array in order to avoid a collection modified exception
            var keys = new string[this.Data.Keys.Count];
            this.Data.Keys.CopyTo(keys, 0);

            foreach (var key in keys)
            {
                //Iterate through each expression in order to check whether the current key must be encrypted and is encrypted.
                //If not then encrypt the value and overwrite the key
                var value = this.Data[key];
                if (!string.IsNullOrEmpty(value) && expressions.Any(e => e.IsMatch(key)))
                {
                    this.encryptedKeys.Add(key);

                    //Verify whether the value is encrypted
                    if (!this.isEncrypted(value))
                    {
                        var protectedValue = ProtectedData.Protect(Encoding.UTF8.GetBytes(value), protectedSource.Entropy, protectedSource.Scope);
                        var protectedValueWithPrefix = new List<byte>(encryptedPrefixBytes);
                        protectedValueWithPrefix.AddRange(protectedValue);

                        //Convert the protected value to a base-64 string in order to mask the prefix (for cosmetic purposes)
                        //and overwrite the key with the encrypted value
                        var protectedBase64Value = Convert.ToBase64String(protectedValueWithPrefix.ToArray());
                        encryptedKeyValuesToWrite.Add(key, protectedBase64Value);
                        this.Data[key] = protectedBase64Value;
                    }
                }
            }

            //Write the encrypted key/values to the JSON configuration file
            this.writeValues(encryptedKeyValuesToWrite);
        }
    }

    /// <summary>Attempts to get the value of the given key</summary>
    /// <param name="key">Key to get</param>
    /// <param name="value">Value of the key</param>
    /// <returns>Returns true in case the key has been found</returns>
    public override bool TryGet(string key, out string value)
    {
        if (!base.TryGet(key, out value)) { return false; }
        else if (!this.encryptedKeys.Contains(key)) { return true; }

        //Key is encrypted and must therefore be decrypted in order to return.
        //Note that the decoded base-64 bytes contains the encrypted prefix which must be excluded when unprotection
        var protectedValueWithPrefix = Convert.FromBase64String(value);
        var protectedValue = new byte[protectedValueWithPrefix.Length - encryptedPrefixBytes.Length];
        Buffer.BlockCopy(protectedValueWithPrefix, encryptedPrefixBytes.Length, protectedValue, 0, protectedValue.Length);

        var unprotectedValue = ProtectedData.Unprotect(protectedValue, this.protectedSource.Entropy, this.protectedSource.Scope);
        value = Encoding.UTF8.GetString(unprotectedValue);
        return true;
    }

/// <summary>Provides extensions concerning <see cref="ProtectedJsonConfigurationProvider"/></summary>
public static class ProtectedJsonConfigurationProviderExtensions
{
    /// <summary>Adds a protected JSON file</summary>
    /// <param name="configurationBuilder"><see cref="IConfigurationBuilder"/> in which to apply the JSON file</param>
    /// <param name="path">Path to the JSON file</param>
    /// <param name="optional">Specifies whether the JSON file is optional</param>
    /// <param name="entropy">Byte array to increase protection</param>
    /// <returns>Returns the <see cref="IConfigurationBuilder"/></returns>
    /// <exception cref="ArgumentNullException"/>
    public static IConfigurationBuilder AddProtectedJsonFile(this IConfigurationBuilder configurationBuilder, string path, bool optional, byte[] entropy, params Regex[] encryptedKeyExpressions)
    {
        var source = new ProtectedJsonConfigurationSource(entropy)
        {
            Path = path,
            Optional = optional,
            EncryptedKeyExpressions = encryptedKeyExpressions
        };

        return configurationBuilder.Add(source);
    }
}

Solution 5

public static IServiceCollection ConfigureProtected<TOptions>(this IServiceCollection services, IConfigurationSection section) where TOptions: class, new()
{
    return services.AddSingleton(provider =>
    {
        var dataProtectionProvider = provider.GetRequiredService<IDataProtectionProvider>();
        var protectedSection = new ProtectedConfigurationSection(dataProtectionProvider, section);

        var options = protectedSection.Get<TOptions>();
        return Options.Create(options);
    });
}

This method is correct

View more solutions

Comments

[paradise_human]

I have same problem. can you complete UnencryptMyConfiguration based on your comment ?(for example load the file and unencrypt key by key)

[CoderSteve]

I posted a solution below that works for me, a lot simpler and perfect for my needs.

[ashilon]

Sure helped me. Thanks a lot.

[ssug89]

Which encryption method will be better??

[blogs4t]

Helped me. Thanks!

[Tseng]

Not really helpful neither does it increase the security a lot. In order to decrypt the data, you need the decryption key, which is within your application. So one attacker who gets on your server will have both, the configuration file and the application and can still easily decrypt your data. You should rather use the DataProtection for local or Azure Key Valut for azure applications.

[Tseng]

These protect you even when your application / local files are stolen, because the private keys to decrypt (data protection) are marked as not-exportable (so an attacker can't easily steal them) and on azure key vault they are outside the application, so when someone copies your appsettings.json and application data from the server, he can't get the user secrets/credentials. The post here is nothing else than fake security (security by obscurity) and offers no protection at all

[CoderSteve]

@Tseng You are correct, but this solution was made prior to when Asp.net Core 2 existed.

[Tseng]

Both Azure KeyVault (.NET Core 1.1) and DataProtection (.NET Core 1.0) where there for a long time, with dataprotection dating back to the rc versions of 1.0

[Thaldin]

I am leveraging this technique. I get an encrypted value from the controller, but then when the protected section attempts to Unprotect, I get a cryptographic exception saying the payload is invalid. oops, hit enter too soon.. Does something special need to be done with the encrypt portion to relate it to the unprotect portion?

[Thaldin]

Looks like I am having a naming issue. When I changed both the DataProtectors naming to "Test", it worked. Something about my section names isn't working, but at least I found the issue.

[Eric Milliot-Martinez]

DataProtection class isn't FIPS compliant, so if you're a government organization, try again. From Microsoft: "meaning only classes that end in CryptoServiceProvider" or "Cng" can be used." docs.microsoft.com/en-us/windows/security/threat-protection/‌​…

[Eric Milliot-Martinez]

Is it FIPS compliant?

[Maverick]

I'm able to encrypt the connection string, but how to I decrypt it again during application startup ?

[Vlad Radu]

By default, a new key is regenerated every 90 days, so that the ciphertext in the configuration file will fail to decrypt again if it is not updated. Also, ASP.NET Core generates different keys to encrypt data when it is running on different machines so it's not usable like this if your app is distributed.

[Vlad Radu]

@Tseng If an attacker owns your server, there's no difference in keeping your secrets in plain text or using Data Protection or Azure Key Vault because he can always do a memory dump of your app to retrieve the data, once your app gets it decrypted.

[Tseng]

@VladRadu: Nowhere claimed it protects where attacker has full access to the server (i.e. can run code and admin/root priviledges). A lot of attacks however consists of unprotected path traversal attacks, i.e. allowing a person to read the configuration file or download the application binaries, but not executing code. In this cases Azure VaultKey does indeed protect your data. Not to speak its easier to manage in VaultKey. One change, affects all applications and instances, where encrypting values in appsettings require you to recompile/redeploy your app and encode its values first

[Juan]

And you may not want to send the text to encrypt in the URL over http

[Karthik D V]

Can you share the working copy of it? for some reason i'm unable to make it work.

[JustAMartin]

I'm also curious, in what form are the secrets stored in PersistKeysToFileSystem specified path? If they are encrypted, then how?

[JustAMartin]

Answering myself - ProtectKeysWithDpapi won't work on *nix machines - it's Windows specific. And PersistKeysToFileSystem alone secrets are encrypted with a raw key. There is even a warning in the XML file.

[sandeep talabathula]

anybody knew whether or not DataProtectionAPI works in PCF? asking because the process is erroring out in accessing the machine key while decrypting.

[Mike Bruno]

For the benefit of someone who is extremely new to ASP.NET Core, how would I ultimately access the connection string value in Startup.ConfigureServices() in order to pass it into UseSqlServerStorage()?