Reading file input from a multipart/form-data POST

Question

I'm POSTing a file to a WCF REST service through a HTML form, with enctype set to multipart/form-data and a single component: <input type="file" name="data">. The resulting stream being read by the server contains the following:

------WebKitFormBoundary
Content-Disposition: form-data; name="data"; filename="DSCF0001.JPG"
Content-Type: image/jpeg

<file bytes>
------WebKitFormBoundary--

The problem is that I'm not sure how do extract the file bytes from the stream. I need to do this in order to write the file to the disk.

Solution 1

You may take a look at the following blog post which illustrates a technique that could be used to parse multipart/form-data on the server using the Multipart Parser:

public void Upload(Stream stream)
{
    MultipartParser parser = new MultipartParser(stream);
    if (parser.Success)
    {
        // Save the file
        SaveFile(parser.Filename, parser.ContentType, parser.FileContents);
    }
}

Another possibility is to enable aspnet compatibility and use HttpContext.Current.Request but that's not a very WCFish way.

Solution 2

Sorry for joining the party late, but there is a way to do this with Microsoft public API.

Here's what you need:

1. System.Net.Http.dll
   • Included in .NET 4.5
   • For .NET 4 get it via NuGet
2. System.Net.Http.Formatting.dll
   • For .NET 4.5 get this NuGet package
   • For .NET 4 get this NuGet package

Note The Nuget packages come with more assemblies, but at the time of writing you only need the above.

Once you have the assemblies referenced, the code can look like this (using .NET 4.5 for convenience):

public static async Task ParseFiles(
    Stream data, string contentType, Action<string, Stream> fileProcessor)
{
    var streamContent = new StreamContent(data);
    streamContent.Headers.ContentType = MediaTypeHeaderValue.Parse(contentType);

    var provider = await streamContent.ReadAsMultipartAsync();

    foreach (var httpContent in provider.Contents)
    {
        var fileName = httpContent.Headers.ContentDisposition.FileName;
        if (string.IsNullOrWhiteSpace(fileName))
        {
            continue;
        }

        using (Stream fileContents = await httpContent.ReadAsStreamAsync())
        {
            fileProcessor(fileName, fileContents);
        }
    }
}

As for usage, say you have the following WCF REST method:

[OperationContract]
[WebInvoke(Method = WebRequestMethods.Http.Post, UriTemplate = "/Upload")]
void Upload(Stream data);

You could implement it like so

public void Upload(Stream data)
{
    MultipartParser.ParseFiles(
           data, 
           WebOperationContext.Current.IncomingRequest.ContentType, 
           MyProcessMethod);
}

Solution 3

I've had some issues with parser that are based on string parsing particularly with large files I found it would run out of memory and fail to parse binary data.

To cope with these issues I've open sourced my own attempt at a C# multipart/form-data parser here

Features:

• Handles very large files well. (Data is streamed in and streamed out while reading)
• Can handle multiple file uploads and automatically detects if a section is a file or not.
• Returns files as a stream not as a byte[] (good for large files).
• Full documentation for the library including a MSDN-style generated website.
• Full unit tests.

Restrictions:

• Doesn't handle non-multipart data.
• Code is more complicated then Lorenzo's

Just use the MultipartFormDataParser class like so:

Stream data = GetTheStream();

// Boundary is auto-detected but can also be specified.
var parser = new MultipartFormDataParser(data, Encoding.UTF8);

// The stream is parsed, if it failed it will throw an exception. Now we can use
// your data!

// The key of these maps corresponds to the name field in your
// form
string username = parser.Parameters["username"].Data;
string password = parser.Parameters["password"].Data

// Single file access:
var file = parser.Files.First();
string filename = file.FileName;
Stream data = file.Data;

// Multi-file access
foreach(var f in parser.Files)
{
    // Do stuff with each file.
}

In the context of a WCF service you could use it like this:

public ResponseClass MyMethod(Stream multipartData)
{
    // First we need to get the boundary from the header, this is sent
    // with the HTTP request. We can do that in WCF using the WebOperationConext:
    var type = WebOperationContext.Current.IncomingRequest.Headers["Content-Type"];

    // Now we want to strip the boundary out of the Content-Type, currently the string
    // looks like: "multipart/form-data; boundary=---------------------124123qase124"
    var boundary = type.Substring(type.IndexOf('=')+1);

    // Now that we've got the boundary we can parse our multipart and use it as normal
    var parser = new MultipartFormDataParser(data, boundary, Encoding.UTF8);

    ...
}

Or like this (slightly slower but more code friendly):

public ResponseClass MyMethod(Stream multipartData)
{
    var parser = new MultipartFormDataParser(data, Encoding.UTF8);
}

Documentation is also available, when you clone the repository simply navigate to HttpMultipartParserDocumentation/Help/index.html

Solution 4

I open-sourced a C# Http form parser here.

This is slightly more flexible than the other one mentioned which is on CodePlex, since you can use it for both Multipart and non-Multipart form-data, and also it gives you other form parameters formatted in a Dictionary object.

This can be used as follows:

non-multipart

public void Login(Stream stream)
{
    string username = null;
    string password = null;

    HttpContentParser parser = new HttpContentParser(stream);
    if (parser.Success)
    {
        username = HttpUtility.UrlDecode(parser.Parameters["username"]);
        password = HttpUtility.UrlDecode(parser.Parameters["password"]);
    }
}

multipart

public void Upload(Stream stream)
{
    HttpMultipartParser parser = new HttpMultipartParser(stream, "image");

    if (parser.Success)
    {
        string user = HttpUtility.UrlDecode(parser.Parameters["user"]);
        string title = HttpUtility.UrlDecode(parser.Parameters["title"]);

        // Save the file somewhere
        File.WriteAllBytes(FILE_PATH + title + FILE_EXT, parser.FileContents);
    }
}

Solution 5

Another way would be to use .Net parser for HttpRequest. To do that you need to use a bit of reflection and simple class for WorkerRequest.

First create class that derives from HttpWorkerRequest (for simplicity you can use SimpleWorkerRequest):

public class MyWorkerRequest : SimpleWorkerRequest
{
    private readonly string _size;
    private readonly Stream _data;
    private string _contentType;

    public MyWorkerRequest(Stream data, string size, string contentType)
        : base("/app", @"c:\", "aa", "", null)
    {
        _size = size ?? data.Length.ToString(CultureInfo.InvariantCulture);
        _data = data;
        _contentType = contentType;
    }

    public override string GetKnownRequestHeader(int index)
    {
        switch (index)
        {
            case (int)HttpRequestHeader.ContentLength:
                return _size;
            case (int)HttpRequestHeader.ContentType:
                return _contentType;
        }
        return base.GetKnownRequestHeader(index);
    }

    public override int ReadEntityBody(byte[] buffer, int offset, int size)
    {
        return _data.Read(buffer, offset, size);
    }

    public override int ReadEntityBody(byte[] buffer, int size)
    {
        return ReadEntityBody(buffer, 0, size);
    }
}

Then wherever you have you message stream create and instance of this class. I'm doing it like that in WCF Service:

[WebInvoke(Method = "POST",
               ResponseFormat = WebMessageFormat.Json,
               BodyStyle = WebMessageBodyStyle.Bare)]
    public string Upload(Stream data)
    {
        HttpWorkerRequest workerRequest =
            new MyWorkerRequest(data,
                                WebOperationContext.Current.IncomingRequest.ContentLength.
                                    ToString(CultureInfo.InvariantCulture),
                                WebOperationContext.Current.IncomingRequest.ContentType
                );

And then create HttpRequest using activator and non public constructor

var r = (HttpRequest)Activator.CreateInstance(
            typeof(HttpRequest),
            BindingFlags.Instance | BindingFlags.NonPublic,
            null,
            new object[]
                {
                    workerRequest,
                    new HttpContext(workerRequest)
                },
            null);

var runtimeField = typeof (HttpRuntime).GetField("_theRuntime", BindingFlags.Static | BindingFlags.NonPublic);
if (runtimeField == null)
{
    return;
}

var runtime = (HttpRuntime) runtimeField.GetValue(null);
if (runtime == null)
{
    return;
}

var codeGenDirField = typeof(HttpRuntime).GetField("_codegenDir", BindingFlags.Instance | BindingFlags.NonPublic);
if (codeGenDirField == null)
{
    return;
}

codeGenDirField.SetValue(runtime, @"C:\MultipartTemp");

After that in r.Files you will have files from your stream.

View more solutions

Comments

[rafale]

What's the matter with using aspnet compatibility?

[Darin Dimitrov]

@rafale, the matter is that of tomorrow you decide to host your WCF service in something else than IIS (like a Windows Service, or whatever) you will have to rewrite it. Other than that nothing wrong.

[rafale]

I forgot to mention that my WCF service is being hosted outside of an IIS as a Managed Windows Service. I'm guessing this means aspnet compatibility isn't available to me either way, correct?

[Darin Dimitrov]

@rafale, absolutely, no aspnet compatibility if you host in a Windows Service. So you will have to parse this multipart/form-data stream and I am afraid there is nothing built-in .NET that might help you. A third party parser is probably your best bet.

[rafale]

This is a little off topic, but is the Content-Type: image/jpeg property in the header detected by the browser submitting the form?

[Darin Dimitrov]

@rafale, I don't understand your question. It's browser sending this request to the server. What detection do you mean? It's up to the server to parse the HTTP request.

[rafale]

The server receives Content-Type: image/jpeg as part of the stream (see original post), and I was just wondering where that comes.

[Darin Dimitrov]

@rafale, it's the browser that formats the request like this because the user selected a jpeg file to upload.

[Bluebaron]

usage: var values = new HttpNameValueCollection(ref httplistenercontext); Console.WriteLine(values.Post["username"]); file.Write(values.Files["file"]);

[Lorenzo Polidori]

mine is MIT license so you can modify it as you like.

[guiomie]

What would I need to put at the place of SaveFile(), Im confused there.

[Darin Dimitrov]

@guiomie, that would depend on what you want to do with the file and where you want to save it.

[guiomie]

File.WriteAllBytes(filepath, parser.FileContents) is what I wanted to do (save the file on the disk)

[sqreept]

How much heap does this consume: byte[] data = Misc.ToByteArray(stream); string content = encoding.GetString(data);

[Ohad Schneider]

Multipart parser is LGPL. The one by @Lorenzo Polidori, besides being more complete, is also MIT licensed (far more permissive).

[Ohad Schneider]

As long as a .NET update doesn't stop it from working, this is by far the best solution (ASP.NET is probably more robust and reliable than existing open source implementations, as good as they may be).

[Ohad Schneider]

I've edited your answer with some necessary code (it may crash for certain files otherwise)

[Hoppe]

The link to your project site on the nuget gallery page is broken

[Lukasz S]

could you say a bit more about what kind of files can cause it to crash? and returns that you've put there does it mean that we cannot create request or maybe just r.Files will throw an exception ?

[Ohad Schneider]

The parser has an optimization for large files, where it stores them on disk. I've populated the necessary fields for this optimization to work. You can control thins threshold via the requestLengthDiskThreshold configuration element in your app.config. Another config element you might want to change is maxrequestlength. The return statements could may as well be exceptions, it means our reflection failed and we may very well fail on big files.

[Devela]

Will this method create a dependency with IIS?

[Frank Myat Thu]

Finally it is work, I really really appreciate your great work, thank a lot @Jake Woods.

[nickvans]

That multi-parser is glorious! Thanks!

[Gerard ONeill]

It seems that you need to make the operation an async task too? Is there a way to consume the async method without having the whole chain be async?

[Ohad Schneider]

@GerardONeill you can always do Task.Wait() when you wish do go synchronous. For example: ParseFiles(data, contentType, fileProcessor).Wait(). Not sure why you'd want to do it though...

[Gerard ONeill]

This way I can plug this functionality into existing code without having to change the entire chain of methods. In my particular spot, I'm just learning about all of this (starting with parsing multipart data), and it helps learning if I don't have to learn how to do 12 things before I get one thing to work.

[Gerard ONeill]

I forgot to say Thanks ;). I was just letting you know the issue with the await operator, especially for a newbie.

[Ohad Schneider]

Hey, we've all been there :) When you're ready to tackle it, I recommend: google.com/…

[SHASHIDHAR MANCHUKONDA]

Hi while parsing I am seeing content type in parsed value :(

[SHASHIDHAR MANCHUKONDA]

@Lorenzo I am not getting exact value , it is combined with headers also

[Bartek Eborn]

Worth noting, that this parser is also available via Nuget: HttpMultipartParser. (in case someone, like me, just did ctrl+f looking for nuget :))

[HaLeiVi]

Is it not possible to reference System.Web stuff from ASP.NET, hand over the inputStream to its components and retrieve parsed fields?

[Rafael Herscovici]

should have been a comment

[Joseph Moore]

The file parser doesn't work. This library works github.com/Http-Multipart-Data-Parser/….