Stream Large File to Web Service

Question

We have a Web service (asmx) that takes a byte array buffer and streams it to an external file service for storage and archiving. It's called by a Windows service that generates fairly small files (3-5M), so creating a byte array that size and feeding it as a param to the web service call has worked well up until now.

I've been tasked with adding a new job to the file queue that will potentially generate files of >70M. Obviously a byte[] that size will overwhelm system memory, so I've been casting about for solutions. The web service has an internal method that takes a FileStream as a param instead of a byte[], so I tried surfacing the FileStream method as a WebMethod. I've updated the references in the Windows service, but a strange thing happens: The FileStream parameter has a namespace specifier attached (something like CallingWindowsService.ExternalWebServiceFileManager.FileStream) and this specified object takes no file name as a constructor, so I can't open a specific file.

I'm totally at sea as to how to approach this. Has anyone else done this - streaming large amounts of data to a web service - and if so, what's the best method? My web service will need either a byte[] or FileStream.

Looking through some of the other messages, it looks like MTOM (not familiar with it) might be a solution, but I'm not sure how to set it up in my web methods or get it working.

Solution 1

You could try a few things, you dont mention what protocols your using or how your hosting it so I will assume it could be IIS7 and your using soap. In your web.config file of the web service you can add the following which will increase the file size allowable for transfer without the 404 error:

  <system.web>
     <httpRuntime executionTimeout="999999" maxRequestLength="2097151" />
     ...
  </system.web>

The second thing you might want to do to the web service's web.config file to permit large file transfers:

  <system.webServer>
    <security>
      <requestFiltering>
        <requestLimits maxAllowedContentLength="2000000000" />
      </requestFiltering>
    </security>
  </system.webServer>

Another possibility:

<location path="Copy.asmx"> <!-- Name of you asmx -->
    <system.webServer>
      <security>
        <requestFiltering>
          <requestLimits maxAllowedContentLength="104857600"/> <!-- 100 megs -->
        </requestFiltering>
      </security>
    </system.webServer>
  </location>

The primary problem with sending byte[]'s over web services is that they get put in the SOAP body which is gets encoded as a base 64 string. Encoding files like this grows the size of the file by as much as two thirds in the soap body (ie. a 6 MB file becomes a 9 MB file over the wire).

One other possibility is "chunking" splitting your data into smaller segments before transmission which may be all you need. The chunkSize (set to 500KB) can be a key factor to improve performance of the upload based upon the factors like network speed, server resources etc.

/// <summary>
/// Chunk the file and upload
/// </summary>
/// <param name="filename"></param>
private void UploadVideo(string filename)
{
    #region Vars
    const int chunkSize = 512000;
    byte[] bytes = null;
    int startIndex, endIndex, length, totalChunks;           

    WS.UploadRequest objRequest = new WS.UploadRequest();            
    #endregion

    try
    {
        if (File.Exists(filename))
        {
            using (WS.UploadService objService = new WS.UploadService())
            {
                using (FileStream fs = new FileStream(filename, FileMode.Open, FileAccess.Read, FileShare.Read))
                {
                    //// Calculate total chunks to be sent to service
                    totalChunks = (int)Math.Ceiling((double)fs.Length / chunkSize);

                    //// Set up Upload request object
                    objRequest.FileName = filename;
                    objRequest.FileSize = fs.Length;

                    for (int i = 0; i < totalChunks; i++)
                    {
                        startIndex = i * chunkSize;
                        endIndex = (int)(startIndex + chunkSize > fs.Length ? fs.Length : startIndex + chunkSize);
                        length = endIndex - startIndex;
                        bytes = new byte[length];

                        //// Read bytes from file, and send upload request
                        fs.Read(bytes, 0, bytes.Length);
                        objRequest.FileBytes = bytes;
                        objService.UploadVideo(objRequest);
                    }
                }

            }
        }
    }
    catch (Exception ex)
    {
        MessageBox.Show(string.Format("Error- {0}", ex.Message));
    }

Solution 2

ASMX web services are a legacy technology that should not be used for new development.

One of the features it lacks is support for streaming and large files. In particular, a message will be duplicated in memory up to four times on the way out to the client.

WCF does support true streaming.

Comments

[John Saunders]

-1: "web services" has no problem streaming files. Only the legacy ASMX web services have that problem.

[Jim Stanley]

The good news is that we've decided to add the capability to an existing WCF service, so the receiving end is taken care of. I'm still at sea, though, about calling the service from my Windows service. The function that calls the file transmission service takes a byte[] as a parameter, so my file creation routine has to return that buffer. It sounds like I need to refactor so that the buffer is declared in the file creation routine and that routine calls the service iteratively to transmit the file in chunks?

[John Saunders]

Maybe refactor everything to use Streams instead of byte arrays? MemoryStream can be constructed from a byte array.