How can you upload files as a stream in go?

Question

There are a number of tutorials about posting files using http.Request in go, but almost invariably they start like this:

file, err := os.Open(path)
if err != nil {
    return nil, err
}
fileContents, err := ioutil.ReadAll(file)

Which is to say, you read the entire file into memory, and then convert it into a Buffer and pass that into a request, something like this:

func send(client *http.Client, file *os.File, endpoint string) {
    body := &bytes.Buffer{}
    io.Copy(body, file)
    req, _ := http.NewRequest("POST", endpoint, body)
    resp, _ := client.Do(req)
}

If you wanted to post a massive file and avoid reading it into memory, but instead steam the file up in chunks... how would you do that?

Solution 1

If you need to set Content-Length, it can be done manually. The following snippet is an example of uploading file and extra parameters as a stream (the code based on Buffer-less Multipart POST in Golang)

//NOTE: for simplicity, error check is omitted
func uploadLargeFile(uri, filePath string, chunkSize int, params map[string]string) {
    //open file and retrieve info
    file, _ := os.Open(filePath)
    fi, _ := file.Stat()
    defer file.Close()    

    //buffer for storing multipart data
    byteBuf := &bytes.Buffer{}

    //part: parameters
    mpWriter := multipart.NewWriter(byteBuf)
    for key, value := range params {
        _ = mpWriter.WriteField(key, value)
    }

    //part: file
    mpWriter.CreateFormFile("file", fi.Name())
    contentType := mpWriter.FormDataContentType()

    nmulti := byteBuf.Len()
    multi := make([]byte, nmulti)
    _, _ = byteBuf.Read(multi)    

    //part: latest boundary
    //when multipart closed, latest boundary is added
    mpWriter.Close()
    nboundary := byteBuf.Len()
    lastBoundary := make([]byte, nboundary)
    _, _ = byteBuf.Read(lastBoundary)

    //calculate content length
    totalSize := int64(nmulti) + fi.Size() + int64(nboundary)
    log.Printf("Content length = %v byte(s)\n", totalSize)

    //use pipe to pass request
    rd, wr := io.Pipe()
    defer rd.Close()

    go func() {
        defer wr.Close()

        //write multipart
        _, _ = wr.Write(multi)

        //write file
        buf := make([]byte, chunkSize)
        for {
            n, err := file.Read(buf)
            if err != nil {
                break
            }
            _, _ = wr.Write(buf[:n])
        }        
        //write boundary
        _, _ = wr.Write(lastBoundary)        
    }()

    //construct request with rd
    req, _ := http.NewRequest("POST", uri, rd)
    req.Header.Set("Content-Type", contentType)
    req.ContentLength = totalSize

    //process request
    client := &http.Client{}
    resp, err := client.Do(req)
    if err != nil {
        log.Fatal(err)
    } else {
        log.Println(resp.StatusCode)
        log.Println(resp.Header)

        body := &bytes.Buffer{}
        _, _ = body.ReadFrom(resp.Body)
        resp.Body.Close()
        log.Println(body)
    }
}

Solution 2

Turns out you can actually pass the *File (or any stream-like) object straight into NewRequest.

Notice the caveat however, that NewRequest (as shown here: https://golang.org/src/net/http/request.go?s=21674:21746#L695) won't actually set the ContentLength unless the stream is explicitly one of:

• *bytes.Buffer
• *bytes.Reader
• *strings.Reader

Since *File isn't one of these, the request will be sent without a content length unless you manually set it, which may cause some servers to discard the body of the incoming request, resulting in a body of '' on the server when it appears to have been correctly sent from the go side.

Comments

[ANisus]

*os.File implements the required io.Reader. So you can basically just do req, _ := http.NewRequest("POST", endpoint, file). Try it out! It won't be "in chunks", but you avoid having it all in memory.

[Volker]

The body if a http.Request is a simple io.Reader (a bit simplified). Just make your stream into and io.Reader. How to do this with files depends on the details you want to achieve rate limiting, buffering, retrying, chunked/ranges, etc.)

[JimB]

Most servers will correctly accept messages without a Content-Length, and they should never reject the body silently (though if an API states that Content-Length is required, it's up to the client to follow through). The client will use Transfer-Encoding: chunked, which is only an issue for old HTTP/1.0 servers, or servers that explicitly disallow chunked transfers.

[Rasmus Hansen]

Do you have an example of doing this with multiple files? So all the files are in the same request, and not multiple requests?

[putu]

@RasmusHansen I wrote small form uploader which can handle multiple files with same request. The code is available here.