XMLHttpRequest: Multipart/Related POST with XML and image as payload

Question

I'm trying to POST an image (with Metadata) to Picasa Webalbums from within a Chrome-Extension. Note that a regular post with Content-Type image/xyz works, as I described here. However, I wish to include a description/keywords and the protocol specification describes a multipart/related format with a XML and data part.

I'm getting the Data through HTML5 FileReader and user file input. I retrieve a binary String using

FileReader.readAsBinaryString(file);

Assume this is my callback code once the FileReader has loaded the string:

function upload_to_album(binaryString, filetype, albumid) {

    var method = 'POST';
    var url = 'http://picasaweb.google.com/data/feed/api/user/default/albumid/' + albumid;
    var request = gen_multipart('Title', 'Description', binaryString, filetype);
    var xhr = new XMLHttpRequest();
    xhr.open(method, url, true);
    xhr.setRequestHeader("GData-Version", '3.0');
    xhr.setRequestHeader("Content-Type",  'multipart/related; boundary="END_OF_PART"');
    xhr.setRequestHeader("MIME-version", "1.0");
    // Add OAuth Token
    xhr.setRequestHeader("Authorization", oauth.getAuthorizationHeader(url, method, ''));
    xhr.onreadystatechange = function(data) {
        if (xhr.readyState == 4) {
            // .. handle response
        }
    };
    xhr.send(request);
}   

The gen_multipart function just generates the multipart from the input values and the XML template and produces the exact same output as the specification (apart from ..binary image data..), but for sake of completeness, here it is:

function gen_multipart(title, description, image, mimetype) {
    var multipart = ['Media multipart posting', "   \n", '--END_OF_PART', "\n",
    'Content-Type: application/atom+xml',"\n","\n",
    "<entry xmlns='http://www.w3.org/2005/Atom'>", '<title>', title, '</title>',
    '<summary>', description, '</summary>',
    '<category scheme="http://schemas.google.com/g/2005#kind" term="http://schemas.google.com/photos/2007#photo" />',
    '</entry>', "\n", '--END_OF_PART', "\n",
    'Content-Type:', mimetype, "\n\n",
    image, "\n", '--END_OF_PART--'];
    return multipart.join("");
}

The problem is, that the POST payload differs from the raw image data, and thus leads to a Bad Request (Picasa won't accept the image), although it worked fine when using

xhr.send(file) // With content-type set to file.type

My question is, how do I get the real binary image to include it in the multipart? I assume it is mangled by just appending it to the xml string, but I can't seem to get it fixed.

Note that due to an old bug in Picasa, base64 is not the solution.

Solution

The XMLHttpRequest specification states that the data send using the .send() method is converted to unicode, and encoded as UTF-8.

The recommended way to upload binary data is through FormData API. However, since you're not just uploading a file, but wrapping the binary data within XML, this option is not useful.

The solution can be found in the source code of the FormData for Web Workers Polyfill, which I've written when I encountered a similar problem. To prevent the Unicode-conversion, all data is added to an array, and finally transmitted as an ArrayBuffer. The byte sequences are not touched on transmission, per specification.

The code below is a specific derivative, based on the FormData for Web Workers Polyfill:

function gen_multipart(title, description, image, mimetype) {
    var multipart = [ "..." ].join(''); // See question for the source
    var uint8array = new Uint8Array(multipart.length);
    for (var i=0; i<multipart.length; i++) {
        uint8array[i] = multipart.charCodeAt(i) & 0xff;
    }
    return uint8array.buffer; // <-- This is an ArrayBuffer object!
}

The script becomes more efficient when you use .readAsArrayBuffer instead of .readAsBinaryString:

function gen_multipart(title, description, image, mimetype) {
    image = new Uint8Array(image); // Wrap in view to get data

    var before = ['Media ... ', 'Content-Type:', mimetype, "\n\n"].join('');
    var after = '\n--END_OF_PART--';
    var size = before.length + image.byteLength + after.length;
    var uint8array = new Uint8Array(size);
    var i = 0;

    // Append the string.
    for (; i<before.length; i++) {
        uint8array[i] = before.charCodeAt(i) & 0xff;
    }

    // Append the binary data.
    for (var j=0; j<image.byteLength; i++, j++) {
        uint8array[i] = image[j];
    }

    // Append the remaining string
    for (var j=0; j<after.length; i++, j++) {
        uint8array[i] = after.charCodeAt(j) & 0xff;
    }
    return uint8array.buffer; // <-- This is an ArrayBuffer object!
}

Comments

[oliverguenther]

As stated twice in my post, uploading the image directly without metadata works fine. I explicitly asked for a solution for sending it with metadata.

[oliverguenther]

Great find! Thanks for the detailed analysis and solution :)

[pduncan]

Please note that XMLHttpRequest.send(ArrayBuffer) is deprecated. You should use XMLHttpRequest.send(ArrayBufferView) (take of the .buffer) or the upcoming XMLHttpRequest.sendAsBinary(). See developer.mozilla.org/en-US/docs/Web/API/…

[Rob W]

@pduncan Use return uint8array; instead of return uint8array.buffer;.

[contactmatt]

@RobW Thank you so much for your in-depth answer :)