Logging request/response messages when using HttpClient

Question

I have a method that does a POST like below

var response = await client.PostAsJsonAsync(url, entity);

if (response.IsSuccessStatusCode)
{
        // read the response as strongly typed object
        return await response.Content.ReadAsAsync<T>();
}

My question is how can I obtain the actual JSON that got posted from the entity object. I would like to log the JSON that gets POSTED, so it will be nice to have that without me having to do a json serialize myself.

Solution 1

An example of how you could do this:

Some notes:

• LoggingHandler intercepts the request before it handles it to HttpClientHandler which finally writes to the wire.

• PostAsJsonAsync extension internally creates an ObjectContent and when ReadAsStringAsync() is called in the LoggingHandler, it causes the formatter inside ObjectContent to serialize the object and that's the reason you are seeing the content in json.

Logging handler:

public class LoggingHandler : DelegatingHandler
{
    public LoggingHandler(HttpMessageHandler innerHandler)
        : base(innerHandler)
    {
    }

    protected override async Task<HttpResponseMessage> SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
    {
        Console.WriteLine("Request:");
        Console.WriteLine(request.ToString());
        if (request.Content != null)
        {
            Console.WriteLine(await request.Content.ReadAsStringAsync());
        }
        Console.WriteLine();

        HttpResponseMessage response = await base.SendAsync(request, cancellationToken);

        Console.WriteLine("Response:");
        Console.WriteLine(response.ToString());
        if (response.Content != null)
        {
            Console.WriteLine(await response.Content.ReadAsStringAsync());
        }
        Console.WriteLine();

        return response;
    }
}

Chain the above LoggingHandler with HttpClient:

HttpClient client = new HttpClient(new LoggingHandler(new HttpClientHandler()));
HttpResponseMessage response = client.PostAsJsonAsync(baseAddress + "/api/values", "Hello, World!").Result;

Output:

Request:
Method: POST, RequestUri: 'http://kirandesktop:9095/api/values', Version: 1.1, Content: System.Net.Http.ObjectContent`1[
[System.String, mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089]], Headers:
{
  Content-Type: application/json; charset=utf-8
}
"Hello, World!"

Response:
StatusCode: 200, ReasonPhrase: 'OK', Version: 1.1, Content: System.Net.Http.StreamContent, Headers:
{
  Date: Fri, 20 Sep 2013 20:21:26 GMT
  Server: Microsoft-HTTPAPI/2.0
  Content-Length: 15
  Content-Type: application/json; charset=utf-8
}
"Hello, World!"

Solution 2

See http://mikehadlow.blogspot.com/2012/07/tracing-systemnet-to-debug-http-clients.html

To configure a System.Net listener to output to both the console and a log file, add the following to your assembly configuration file:

<system.diagnostics>
  <trace autoflush="true" />
  <sources>
    <source name="System.Net">
      <listeners>
        <add name="MyTraceFile"/>
        <add name="MyConsole"/>
      </listeners>
    </source>
  </sources>
  <sharedListeners>
    <add
      name="MyTraceFile"
      type="System.Diagnostics.TextWriterTraceListener"
      initializeData="System.Net.trace.log" />
    <add name="MyConsole" type="System.Diagnostics.ConsoleTraceListener" />
  </sharedListeners>
  <switches>
    <add name="System.Net" value="Verbose" />
  </switches>
</system.diagnostics>

Solution 3

Network tracing also available for next objects (see article on msdn)

• System.Net.Sockets Some public methods of the Socket, TcpListener, TcpClient, and Dns classes
• System.Net Some public methods of the HttpWebRequest, HttpWebResponse, FtpWebRequest, and FtpWebResponse classes, and SSL debug information (invalid certificates, missing issuers list, and client certificate errors.)
• System.Net.HttpListener Some public methods of the HttpListener, HttpListenerRequest, and HttpListenerResponse classes.
• System.Net.Cache Some private and internal methods in System.Net.Cache.
• System.Net.Http Some public methods of the HttpClient, DelegatingHandler, HttpClientHandler, HttpMessageHandler, MessageProcessingHandler, and WebRequestHandler classes.
• System.Net.WebSockets.WebSocket Some public methods of the ClientWebSocket and WebSocket classes.

Put the following lines of code to the configuration file:

<configuration>  
  <system.diagnostics>  
    <sources>  
      <source name="System.Net" tracemode="includehex" maxdatasize="1024">  
        <listeners>  
          <add name="System.Net"/>  
        </listeners>  
      </source>  
      <source name="System.Net.Cache">  
        <listeners>  
          <add name="System.Net"/>  
        </listeners>  
      </source>  
      <source name="System.Net.Http">  
        <listeners>  
          <add name="System.Net"/>  
        </listeners>  
      </source>  
      <source name="System.Net.Sockets">  
        <listeners>  
          <add name="System.Net"/>  
        </listeners>  
      </source>  
      <source name="System.Net.WebSockets">  
        <listeners>  
          <add name="System.Net"/>  
        </listeners>  
      </source>  
    </sources>  
    <switches>  
      <add name="System.Net" value="Verbose"/>  
      <add name="System.Net.Cache" value="Verbose"/>  
      <add name="System.Net.Http" value="Verbose"/>  
      <add name="System.Net.Sockets" value="Verbose"/>  
      <add name="System.Net.WebSockets" value="Verbose"/>  
    </switches>  
    <sharedListeners>  
      <add name="System.Net"  
        type="System.Diagnostics.TextWriterTraceListener"  
        initializeData="network.log"  
      />  
    </sharedListeners>  
    <trace autoflush="true"/>  
  </system.diagnostics>  
</configuration>  

Comments

[mathk]

That is nice if you need the request details but it fail at getting the exact request sent to the server. If you need precisely all the byte sent to the server it not gonna work this way.

[Brian Oliver]

.NET adds Expect: 100-continue to the headers of POST requests, and this method will not show that. To get rid of that particular header, do this: client.DefaultRequestHeaders.ExpectContinue = false;

[Y.Doktur]

Imagine a world where most of these type of connections are actually HTTPS.

[Lars335]

@kiran-challa: You might want to add .ConfigureAwait(false) to the await-lines to prevent possible deadlocks: msdn.microsoft.com/en-us/magazine/…

[Zero3]

Why the new HttpClientHandler()? It is not present in the official docs: docs.microsoft.com/en-us/aspnet/web-api/overview/advanced/…

[Zero3]

Ah, it is apparently required to not get an exception about the inner handler being null...

[IllusiveBrian]

You can also override MessageProcessingHandler which basically calls a ProcessRequest and ProcessResponse method for you before and after the SendAsync call.

[Tsahi Asher]

@RamiA's answer below is better, because it doesn't require code changes. Once you're done debugging, you remove the tracing from your config, and that's it. No need to make a new build.

[Vitor Paulino]

Consider adding a dependency to ILoggerProvider from Microsoft.Extensions.Logging that way you delegate the sink that you are going to use in configuration. Then you can use any .net library that exists outthere that implements this contracts from Microsoft

[Muflix]

How does it work ? I copied <system.diagnostics> into app.config under <configuration> but in bin folder there is no log file and console output don't display anything too, what am I missing ?

[Rami A.]

@Muflix, you can read the documentation which is linked in the page referenced at the top of my answer. I think the file name specified in the initializeData attribute will be created in the current working directory of the executable you are running, so you may want to check which path that is in your environment.

[Brayan Loayza]

Where is the configuration file, I cannot find it.

[The incredible Jan]

@Rami A. Nobody would seriously call the linked blog post "documentation". It doesn't explain more then your answer. For me it's not working but maybe for a plugin dll I have to do something more... or just write the interesting stuff to file by myself like Kiran Challa does.

[Rami A.]

@TheincredibleJan there's a link within the blogpost: docs.microsoft.com/en-us/dotnet/framework/network-programmin‌​g/… You can contribute to the documentation by providing your feedback at the bottom of the page. It's the official documentation and contains more technical information than my answer. Why don't you think it's "serious" documentation?

[EpsilonDelta]

When I read and log the stream like this, then I cannot read and process the stream in the returned response. How can I solve this problem? I want to log it, but I don't want to mess with the response object (and it's content stream) in any way.

[Squirrelkiller]

Depending on your project type either web.config, app.config or possibly even appsettings.json in which case you'll have to figure out these options in json.

[Dave Lawrence]

@TsahiAsher It's not better. It's just different. What the preferable solution is depends on the use case. For instance it will often be a key requirement of an enterprise application which integrates with an external third party that every engagement with that third party is audited and persisted. Dumping to trace files is not necessarily a sustainable solution.

[Tsahi Asher]

@DaveLawrence you can dump it to anything. There are several listeners built in, and you can write your own or use a third party listener which will send it to a logging framework. The possibilities are endless.