Decompressing GZip Stream from HTTPClient Response

Question

I am trying to connect to an api, that returns GZip encoded JSON, from a WCF service (WCF service to WCF service). I am using the HTTPClient to connect to the API and have been able to return the JSON object as a string. However I need to be able to store this returned data in a database and as such I figured the best way would be to return and store the JSON object in an array or byte or something along those lines.

What I am having trouble with specifically is the decompressing of the GZip encoding and have been trying lots of different example but still cant get it.

The below code is how I am establishing my connection and getting a response, this is the code that returns a string from the API.

public string getData(string foo)
{
    string url = "";
    HttpClient client = new HttpClient();
    HttpResponseMessage response;
    string responseJsonContent;
    try
    {
        client.DefaultRequestHeaders.Accept.Add(new MediaTypeWithQualityHeaderValue("application/json"));
        response = client.GetAsync(url + foo).Result;
        responseJsonContent = response.Content.ReadAsStringAsync().Result;
        return responseJsonContent;
    }
    catch (Exception ex)
    {
        System.Windows.Forms.MessageBox.Show(ex.Message);
        return "";
    }
}

I have been following a few different examples like these StackExchange API, MSDN, and a couple on stackoverflow, but I haven't been able to get any of these to work for me.

What is the best way to accomplish this, am I even on the right track?

Thanks guys.

Solution 1

Just instantiate HttpClient like this:

HttpClientHandler handler = new HttpClientHandler()
{
    AutomaticDecompression = DecompressionMethods.GZip | DecompressionMethods.Deflate
};

using (var client = new HttpClient(handler)) //see update below
{
    // your code
}

Update June 19, 2020: It's not recommended to use httpclient in a 'using' block as it might cause port exhaustion.

private static HttpClient client = null;
    
ContructorMethod()
{
   if(client == null)
   {
        HttpClientHandler handler = new HttpClientHandler()
        {
            AutomaticDecompression = DecompressionMethods.GZip | DecompressionMethods.Deflate
        };        
        client = new HttpClient(handler);
   }
// your code            
 }

If using .Net Core 2.1+, consider using IHttpClientFactory and injecting like this in your startup code.

 var timeout = Policy.TimeoutAsync<HttpResponseMessage>(
            TimeSpan.FromSeconds(60));

 services.AddHttpClient<XApiClient>().ConfigurePrimaryHttpMessageHandler(() => new HttpClientHandler
        {
            AutomaticDecompression = DecompressionMethods.GZip | DecompressionMethods.Deflate
        }).AddPolicyHandler(request => timeout);

Solution 2

I used code from below link to decompress GZip stream.Then used the decompressed byte array to get the required JSON object. Hope it may help some one.

var readTask = result.Content.ReadAsByteArrayAsync().Result;
var decompressedData = Decompress(readTask);
string jsonString = System.Text.Encoding.UTF8.GetString(decompressedData, 0, decompressedData.Length);
ResponseObjectClass responseObject = Newtonsoft.Json.JsonConvert.DeserializeObject<ResponseObjectClass>(jsonString);

https://www.dotnetperls.com/decompress

static byte[] Decompress(byte[] gzip)
{
    using (GZipStream stream = new GZipStream(new MemoryStream(gzip), CompressionMode.Decompress))
    {
        const int size = 4096;
        byte[] buffer = new byte[size];
        using (MemoryStream memory = new MemoryStream())
        {
            int count = 0;
            do
            {
                count = stream.Read(buffer, 0, size);
                if (count > 0)
                {
                    memory.Write(buffer, 0, count);
                }
            }
            while (count > 0);
            return memory.ToArray();
        }
    }
}

Solution 3

Ok so I eventually solved my problem. If there are better ways please let me know :-)

        public DataSet getData(string strFoo)
    {
        string url = "foo";

        HttpClient client = new HttpClient();
        HttpResponseMessage response;   
        DataSet dsTable = new DataSet();
        try
        {
               //Gets the headers that should be sent with each request
            client.DefaultRequestHeaders.Accept.Add(new MediaTypeWithQualityHeaderValue("application/json"));
              //Returned JSON
            response = client.GetAsync(url).Result;
              //converts JSON to string
            string responseJSONContent = response.Content.ReadAsStringAsync().Result;
              //deserializes string to list
            var jsonList = DeSerializeJsonString(responseJSONContent);
              //converts list to dataset. Bad name I know.
            dsTable = Foo_ConnectAPI.ExtentsionHelpers.ToDataSet<RootObject>(jsonList);
              //Returns the dataset                
            return dsTable;
        }
        catch (Exception ex)
        {
            System.Windows.Forms.MessageBox.Show(ex.Message);
            return null;
        }
    }

       //deserializes the string to a list. Utilizes JSON.net. RootObject is a class that contains the get and set for the JSON elements

    public List<RootObject> DeSerializeJsonString(string jsonString)
    {
          //Initialized the List
        List<RootObject> list = new List<RootObject>();
          //json.net deserializes string
        list = (List<RootObject>)JsonConvert.DeserializeObject<List<RootObject>>(jsonString);

        return list;
    }

The RootObject contains the get set that will get the values of the JSON.

public class RootObject
{  
      //These string will be set to the elements within the JSON. Each one is directly mapped to the JSON elements.
      //This only takes into account a JSON that doesn't contain nested arrays
    public string EntityID { get; set; }

    public string Address1 { get; set; }

    public string Address2 { get; set; }

    public string Address3 { get; set; }

}

The easiest way to create the above class(es) is to use json2charp which will format it accordingly and also provide the correct datatypes.

The following is from another answer on Stackoverflow again it does not take into account nested JSON.

    internal static class ExtentsionHelpers
{
    public static DataSet ToDataSet<T>(this List<RootObject> list)
    {
        try
        {
            Type elementType = typeof(RootObject);
            DataSet ds = new DataSet();
            DataTable t = new DataTable();
            ds.Tables.Add(t);

            try
            {
                //add a column to table for each public property on T
                foreach (var propInfo in elementType.GetProperties())
                {
                    try
                    {
                        Type ColType = Nullable.GetUnderlyingType(propInfo.PropertyType) ?? propInfo.PropertyType;

                            t.Columns.Add(propInfo.Name, ColType);

                    }
                    catch (Exception ex)
                    {
                        System.Windows.Forms.MessageBox.Show(ex.Message);
                    }

                }
            }
            catch (Exception ex)
            {
                System.Windows.Forms.MessageBox.Show(ex.Message);
            }

            try
            {
                //go through each property on T and add each value to the table
                foreach (RootObject item in list)
                {
                    DataRow row = t.NewRow();

                    foreach (var propInfo in elementType.GetProperties())
                    {
                        row[propInfo.Name] = propInfo.GetValue(item, null) ?? DBNull.Value;
                    }

                    t.Rows.Add(row);
                }
            }
            catch (Exception ex)
            {
                System.Windows.Forms.MessageBox.Show(ex.Message);
            }

            insert.insertCategories(t);
            return ds.
        }
        catch (Exception ex)
        {
            System.Windows.Forms.MessageBox.Show(ex.Message);

            return null;
        }
    }
};

Then finally to insert the above dataset into a table with columns that were mapped to the JSON I utilized SQL bulk copy and the following class

public class insert
{ 
    public static string insertCategories(DataTable table)
    {     
        SqlConnection objConnection = new SqlConnection();
          //As specified in the App.config/web.config file
        objConnection.ConnectionString = System.Configuration.ConfigurationManager.ConnectionStrings["foo"].ToString();

        try
        {                                 
            objConnection.Open();
            var bulkCopy = new SqlBulkCopy(objConnection.ConnectionString);

            bulkCopy.DestinationTableName = "dbo.foo";
            bulkCopy.BulkCopyTimeout = 600;
            bulkCopy.WriteToServer(table);

            return "";
        }
        catch (Exception ex)
        {
            System.Windows.Forms.MessageBox.Show(ex.Message);
            return "";
        }
        finally
        {
            objConnection.Close();
        }         
    }
};

So the above works to insert JSON from a webAPI into a database. This is something that I get to work. But by no means do I expect it to be perfect. If you have any improvements then please update it accordingly.

Comments

[user3285954]

"the best way would be to return and store the JSON object in an array or byte" Note that a string is an array of bytes.

[Ian Mercer]

You should create your HttpClient and your HttpResponse inside a using() statement each to ensure proper, timely disposal and closing of the underlying streams.

[FoxDeploy]

If I use this structure, how do I retrieve the content of my response from the httpClient? I'm super new to c# and I don't think I'm getting it.

[DIG]

@FoxDeploy there is no change needed for the code to get the content when you use this solution. See here for reference: stackoverflow.com/questions/26597665/…

[Sebastian Castaldi]

even though it is an old post, this answer just solved my problem in .netcore, moving from 1.1 to 2.0 it seems the client was automatically doing the decompression, so I had to add this code in 2.0 to make it work... Thanks!

[KallDrexx]

Just to piggy back on @SebastianCastaldi, but .net core 1.1 had AutomaticDecompression set properly, but in .net core 2.0 it is set to NONE. This took me way too long to figure out...

[Johnny Oshika]

Is HttpClient unable to decompress Brotli? It seems that DecompressionMethods's only options are None, GZip, and Deflate.

[imba-tjd]

Note: HttpClient Should NOT be used inside using

[Chi Minh Trinh]

I am using it on Xamarin iOS widget and it is crashed. But I don't know why. Could you give me some suggestion @DIG

[jugg1es]

I second @imba-tjd to never use using with HttpClient. You will cause port exhaustion with any real traffic. You should only instantiate HttpClient once during app lifetime if possible.

[jugg1es]

np! I learned the hard way when every VM on a host in our cluster stopped working complaining about ports. It took weeks to figure out it was an app that wasn't using HttpClient correctly.

[JHBonarius]

I must put an emphasis on the upvoted comment HttpClient should NOT be used inside a using block

[Egbert Nierop]

For .NET 5 (not sure about 3) it must be configured like this... (because TimeOut is no longer a policy) services.AddHttpClient<ApiClient>(client => client.Timeout = TimeSpan.FromSeconds( webOptions.HttpTimeout)) .ConfigurePrimaryHttpMessageHandler(() => new HttpClientHandler { AutomaticDecompression = DecompressionMethods.GZip | DecompressionMethods.Brotli } );