How does the paste image from clipboard functionality work in Gmail and Google Chrome 12+?

Question

I noticed a blog post from Google that mentions the ability to paste images directly from the clipboard into a Gmail message if you're using the latest version of Chrome. I tried this with my version of Chrome (12.0.742.91 beta-m) and it works great using control keys or the context menu.

From that behavior I need to assume that the latest version of webkit used in Chrome is able to deal with images in the Javascript paste event, but I have been unable to locate any references to such an enhancement. I believe ZeroClipboard binds to keypress events to trigger its flash functionality and as such wouldn't work through the context menu (also, ZeroClipboard is cross-browser and the post says this works only with Chrome).

So, how does this work and where the enhancement was made to Webkit (or Chrome) that enables the functionality?

Solution 1

I spent some time experimenting with this. It seems to sort of follow the new Clipboard API spec. You can define a "paste" event handler and look at event.clipboardData.items, and call getAsFile() on them to get a Blob. Once you have a Blob, you can use FileReader on it to see what's in it. This is how you can get a data url for the stuff you just pasted in Chrome:

document.onpaste = function (event) {
    var items = (event.clipboardData || event.originalEvent.clipboardData).items;
    console.log(JSON.stringify(items)); // might give you mime types
    for (var index in items) {
        var item = items[index];
        if (item.kind === 'file') {
            var blob = item.getAsFile();
            var reader = new FileReader();
            reader.onload = function (event) {
                console.log(event.target.result); // data url!
            }; 
            reader.readAsDataURL(blob);
        }
    }
};

Once you have a data url you can display the image on the page. If you want to upload it instead, you could use readAsBinaryString, or you could put it into an XHR using FormData.

Edit: Note that the item is of type DataTransferItem. JSON.stringify might not work on the items list, but you should be able to get mime type when you loop over items.

Solution 2

The answer by Nick seems to need small changes to still work :)

// window.addEventListener('paste', ... or
document.onpaste = function (event) {
  // use event.originalEvent.clipboard for newer chrome versions
  var items = (event.clipboardData  || event.originalEvent.clipboardData).items;
  console.log(JSON.stringify(items)); // will give you the mime types
  // find pasted image among pasted items
  var blob = null;
  for (var i = 0; i < items.length; i++) {
    if (items[i].type.indexOf("image") === 0) {
      blob = items[i].getAsFile();
    }
  }
  // load image if there is a pasted image
  if (blob !== null) {
    var reader = new FileReader();
    reader.onload = function(event) {
      console.log(event.target.result); // data url!
    };
    reader.readAsDataURL(blob);
  }
}

Example running code: http://jsfiddle.net/bt7BU/225/

So the changes to nicks answer were:

var items = event.clipboardData.items;

to

var items = (event.clipboardData  || event.originalEvent.clipboardData).items;

Also I had to take the second element from the pasted items (first one seems to be text/html if you copy an image from another web page into the buffer). So I changed

  var blob = items[0].getAsFile();

to a loop finding the item containing the image (see above)

I didn't know how to answer directly to Nick's answer, hope it is fine here :$ :)

Solution 3

As far as I know -

With HTML 5 features(File Api and the related) - accessing clipboard image data is now possible with plain javascript.

This however fails to work on IE (anything less than IE 10). Don't know much about IE10 support also.

For IE the optiens that I believe are the 'fallback' options are either using Adobe's AIR api or using a signed applet

Comments

[Sébastien]

It seems that it works randomly with Firefox as well. Anybody knows if this is supposed to be supported with Firefox ?

[Gavin Gilmour]

Clutching at straws here, but any ideas why event.clipboardData.items seems to be 'undefined' in Safari 5.1? Or even how to get the clipboard contents for a file/blob in Safari? Works great in Chrome. You'd think webkit would be webkit :(

[Nick Retallack]

@SenicaGonzalez that is because the data only exists for the duration of the event. After the event, it is gone, so when you try to flip open the object in the inspector you will see nothing.

[poitroae]

Would you mind making an example how to submit a XMLHttpRequest with that image data? That would be real nice :D

[poitroae]

How should we submit the image data as XMLHttpRequest?

[robintibor]

To others reading this, answer to this question may be included there now: stackoverflow.com/questions/18055422/… :)

[J.T. Taylor]

Here's how you can submit that using XMLHttpRequest, I wrote it up in a blog after I implemented it: blog.securevideo.com/2013/11/27/…

[PixnBits]

@jitbit I believe event.clipboardData has been replaced with event.dataTransfer

[Tomáš Zato]

I don't understand. When I paste files in browser, the clipboardData.items is always empty in google chrome (Firefox works). The chrome now needs almost as much optimization as IE used to.

[Pancakeo]

Small edit: if (blob != null) { (or set blob = null in initialization)

[immayankmodi]

As @TomášZato said It only works in FF (but with error "TypeError: items is undefined") and not working in Chrome, Safari, IE. Is there any updates?

[Nick Retallack]

Now that the first item in the list isn't always the file you pasted, I've updated it to loop through the items to find any pasted files.

[adib]

in iOS' WebKit, clipboardData seems to be an instance of DataTransfer

[Champ]

readAsBinaryString is non-standard and deprecated. Can use readAsArrayBuffer instead. Source: developer.mozilla.org/en-US/docs/Web/API/FileReader/…

[Matthew Moisen]

@MayankModi this works for me in Chrome but not FF :) In FF I get TypeError items is undefined

[1.21 gigawatts]

@MatthewMoisen Instead of the items property check the files property. developer.mozilla.org/en-US/docs/Web/API/ClipboardEvent

[qba]

http://jsfiddle.net/bt7BU/225/ demo doesn't work with newest Chrome 59.0.3071.115 on Win10. Can anybody help?

[Ruben Martinez Jr.]

event.clipboardData.items worked fine for me on the latest Chrome, not sure when event.originalEvent... is useful?

[Peter Bowers]

I believe that the "originalEvent" is needed only if your object is a jQuery object. In other words, if your event is vanilla Javascript then it's not needed; if it's a jQuery object then you need the originalEvent to get to the underlying items.

[Doomer]

But this function does not return anything to work with, nor so it affect any global variable, in other words, this function does nothing and there are no indication of where my file will be especally if I want to use readAsBinaryString

[Max]

Size of the file increases tremendously after this operation. am i missing something?