Using HTML5/Canvas/JavaScript to take in-browser screenshots

Question

Google's "Report a Bug" or "Feedback Tool" lets you select an area of your browser window to create a screenshot that is submitted with your feedback about a bug.

_{Screenshot by Jason Small, posted in a duplicate question.}

How are they doing this? Google's JavaScript feedback API is loaded from here and their overview of the feedback module will demonstrate the screenshot capability.

Solution 1

JavaScript can read the DOM and render a fairly accurate representation of that using canvas. I have been working on a script which converts HTML into a canvas image. Decided today to make an implementation of it into sending feedbacks like you described.

The script allows you to create feedback forms which include a screenshot, created on the client's browser, along with the form. The screenshot is based on the DOM and as such may not be 100% accurate to the real representation as it does not make an actual screenshot, but builds the screenshot based on the information available on the page.

It does not require any rendering from the server, as the whole image is created on the client's browser. The HTML2Canvas script itself is still in a very experimental state, as it does not parse nearly as much of the CSS3 attributes I would want it to, nor does it have any support to load CORS images even if a proxy was available.

Still quite limited browser compatibility (not because more couldn't be supported, just haven't had time to make it more cross browser supported).

For more information, have a look at the examples here:

http://hertzen.com/experiments/jsfeedback/

edit The html2canvas script is now available separately here and some examples here.

edit 2 Another confirmation that Google uses a very similar method (in fact, based on the documentation, the only major difference is their async method of traversing/drawing) can be found in this presentation by Elliott Sprehn from the Google Feedback team: http://www.elliottsprehn.com/preso/fluentconf/

Solution 2

Your web app can now take a 'native' screenshot of the client's entire desktop using getUserMedia():

Have a look at this example:

https://www.webrtc-experiment.com/Pluginfree-Screen-Sharing/

The client will have to be using chrome (for now) and will need to enable screen capture support under chrome://flags.

Solution 3

PoC

As Niklas mentioned you can use the html2canvas library to take a screenshot using JS in the browser. I will extend his answer in this point by providing an example of taking a screenshot using this library ("Proof of Concept"):

function report() {
  let region = document.querySelector("body"); // whole screen
  html2canvas(region, {
    onrendered: function(canvas) {
      let pngUrl = canvas.toDataURL(); // png in dataURL format
      let img = document.querySelector(".screen");
      img.src = pngUrl; 

      // here you can allow user to set bug-region
      // and send it with 'pngUrl' to server
    },
  });
}

.container {
  margin-top: 10px;
  border: solid 1px black;
}

<script src="https://cdnjs.cloudflare.com/ajax/libs/html2canvas/0.4.1/html2canvas.min.js"></script>
<div>Screenshot tester</div>
<button onclick="report()">Take screenshot</button>

<div class="container">
  <img width="75%" class="screen">
</div>

In report() function in onrendered after getting image as data URI you can show it to the user and allow him to draw "bug region" by mouse and then send a screenshot and region coordinates to the server.

In this example async/await version was made: with nice makeScreenshot() function.

UPDATE

Simple example which allows you to take screenshot, select region, describe bug and send POST request (here jsfiddle) (the main function is report()).

async function report() {
    let screenshot = await makeScreenshot(); // png dataUrl
    let img = q(".screen");
    img.src = screenshot; 
    
    let c = q(".bug-container");
    c.classList.remove('hide')
        
    let box = await getBox();    
    c.classList.add('hide');

    send(screenshot,box); // sed post request  with bug image, region and description
    alert('To see POST requset with image go to: chrome console > network tab');
}

// ----- Helper functions

let q = s => document.querySelector(s); // query selector helper
window.report = report; // bind report be visible in fiddle html

async function  makeScreenshot(selector="body") 
{
  return new Promise((resolve, reject) => {  
    let node = document.querySelector(selector);
    
    html2canvas(node, { onrendered: (canvas) => {
        let pngUrl = canvas.toDataURL();      
        resolve(pngUrl);
    }});  
  });
}

async function getBox(box) {
  return new Promise((resolve, reject) => {
     let b = q(".bug");
     let r = q(".region");
     let scr = q(".screen");
     let send = q(".send");
     let start=0;
     let sx,sy,ex,ey=-1;
     r.style.width=0;
     r.style.height=0;
     
     let drawBox= () => {
         r.style.left   = (ex > 0 ? sx : sx+ex ) +'px'; 
         r.style.top    = (ey > 0 ? sy : sy+ey) +'px';
         r.style.width  = Math.abs(ex) +'px';
         r.style.height = Math.abs(ey) +'px'; 
     }
     
     
     
     //console.log({b,r, scr});
     b.addEventListener("click", e=>{
       if(start==0) {
         sx=e.pageX;
         sy=e.pageY;
         ex=0;
         ey=0;
         drawBox();
       }
       start=(start+1)%3;       
     });
     
     b.addEventListener("mousemove", e=>{
       //console.log(e)
       if(start==1) {
           ex=e.pageX-sx;
           ey=e.pageY-sy
           drawBox(); 
       }
     });
     
     send.addEventListener("click", e=>{
       start=0;
       let a=100/75 //zoom out img 75%       
       resolve({
          x:Math.floor(((ex > 0 ? sx : sx+ex )-scr.offsetLeft)*a),
          y:Math.floor(((ey > 0 ? sy : sy+ey )-b.offsetTop)*a),
          width:Math.floor(Math.abs(ex)*a),
          height:Math.floor(Math.abs(ex)*a),
          desc: q('.bug-desc').value
          });
          
     });
  });
}

function send(image,box) {

    let formData = new FormData();
    let req = new XMLHttpRequest();
    
    formData.append("box", JSON.stringify(box)); 
    formData.append("screenshot", image);     
    
    req.open("POST", '/upload/screenshot');
    req.send(formData);
}

.bug-container { background: rgb(255,0,0,0.1); margin-top:20px; text-align: center; }
.send { border-radius:5px; padding:10px; background: green; cursor: pointer; }
.region { position: absolute; background: rgba(255,0,0,0.4); }
.example { height: 100px; background: yellow; }
.bug { margin-top: 10px; cursor: crosshair; }
.hide { display: none; }
.screen { pointer-events: none }

<script src="https://cdnjs.cloudflare.com/ajax/libs/html2canvas/0.4.1/html2canvas.min.js"></script>
<body>
<div>Screenshot tester</div>
<button onclick="report()">Report bug</button>

<div class="example">Lorem ipsum</div>

<div class="bug-container hide">
  <div>Select bug region: click once - move mouse - click again</div>
  <div class="bug">    
    <img width="75%" class="screen" >
    <div class="region"></div> 
  </div>
  <div>
    <textarea class="bug-desc">Describe bug here...</textarea>
  </div>
  <div class="send">SEND BUG</div>
</div>

</body>

Solution 4

Get screenshot as Canvas or Jpeg Blob / ArrayBuffer using getDisplayMedia API:

FIX 1: Use the getUserMedia with chromeMediaSource only for Electron.js
FIX 2: Throw error instead return null object
FIX 3: Fix demo to prevent the error: getDisplayMedia must be called from a user gesture handler

// docs: https://developer.mozilla.org/en-US/docs/Web/API/MediaDevices/getDisplayMedia
// see: https://www.webrtc-experiment.com/Pluginfree-Screen-Sharing/#20893521368186473
// see: https://github.com/muaz-khan/WebRTC-Experiment/blob/master/Pluginfree-Screen-Sharing/conference.js

function getDisplayMedia(options) {
    if (navigator.mediaDevices && navigator.mediaDevices.getDisplayMedia) {
        return navigator.mediaDevices.getDisplayMedia(options)
    }
    if (navigator.getDisplayMedia) {
        return navigator.getDisplayMedia(options)
    }
    if (navigator.webkitGetDisplayMedia) {
        return navigator.webkitGetDisplayMedia(options)
    }
    if (navigator.mozGetDisplayMedia) {
        return navigator.mozGetDisplayMedia(options)
    }
    throw new Error('getDisplayMedia is not defined')
}

function getUserMedia(options) {
    if (navigator.mediaDevices && navigator.mediaDevices.getUserMedia) {
        return navigator.mediaDevices.getUserMedia(options)
    }
    if (navigator.getUserMedia) {
        return navigator.getUserMedia(options)
    }
    if (navigator.webkitGetUserMedia) {
        return navigator.webkitGetUserMedia(options)
    }
    if (navigator.mozGetUserMedia) {
        return navigator.mozGetUserMedia(options)
    }
    throw new Error('getUserMedia is not defined')
}

async function takeScreenshotStream() {
    // see: https://developer.mozilla.org/en-US/docs/Web/API/Window/screen
    const width = screen.width * (window.devicePixelRatio || 1)
    const height = screen.height * (window.devicePixelRatio || 1)

    const errors = []
    let stream
    try {
        stream = await getDisplayMedia({
            audio: false,
            // see: https://developer.mozilla.org/en-US/docs/Web/API/MediaStreamConstraints/video
            video: {
                width,
                height,
                frameRate: 1,
            },
        })
    } catch (ex) {
        errors.push(ex)
    }

    // for electron js
    if (navigator.userAgent.indexOf('Electron') >= 0) {
        try {
            stream = await getUserMedia({
                audio: false,
                video: {
                    mandatory: {
                        chromeMediaSource: 'desktop',
                        // chromeMediaSourceId: source.id,
                        minWidth         : width,
                        maxWidth         : width,
                        minHeight        : height,
                        maxHeight        : height,
                    },
                },
            })
        } catch (ex) {
            errors.push(ex)
        }
    }

    if (errors.length) {
        console.debug(...errors)
        if (!stream) {
            throw errors[errors.length - 1]
        }
    }

    return stream
}

async function takeScreenshotCanvas() {
    const stream = await takeScreenshotStream()

    // from: https://stackoverflow.com/a/57665309/5221762
    const video = document.createElement('video')
    const result = await new Promise((resolve, reject) => {
        video.onloadedmetadata = () => {
            video.play()
            video.pause()

            // from: https://github.com/kasprownik/electron-screencapture/blob/master/index.js
            const canvas = document.createElement('canvas')
            canvas.width = video.videoWidth
            canvas.height = video.videoHeight
            const context = canvas.getContext('2d')
            // see: https://developer.mozilla.org/en-US/docs/Web/API/HTMLVideoElement
            context.drawImage(video, 0, 0, video.videoWidth, video.videoHeight)
            resolve(canvas)
        }
        video.srcObject = stream
    })

    stream.getTracks().forEach(function (track) {
        track.stop()
    })
    
    if (result == null) {
        throw new Error('Cannot take canvas screenshot')
    }

    return result
}

// from: https://stackoverflow.com/a/46182044/5221762
function getJpegBlob(canvas) {
    return new Promise((resolve, reject) => {
        // docs: https://developer.mozilla.org/en-US/docs/Web/API/HTMLCanvasElement/toBlob
        canvas.toBlob(blob => resolve(blob), 'image/jpeg', 0.95)
    })
}

async function getJpegBytes(canvas) {
    const blob = await getJpegBlob(canvas)
    return new Promise((resolve, reject) => {
        const fileReader = new FileReader()

        fileReader.addEventListener('loadend', function () {
            if (this.error) {
                reject(this.error)
                return
            }
            resolve(this.result)
        })

        fileReader.readAsArrayBuffer(blob)
    })
}

async function takeScreenshotJpegBlob() {
    const canvas = await takeScreenshotCanvas()
    return getJpegBlob(canvas)
}

async function takeScreenshotJpegBytes() {
    const canvas = await takeScreenshotCanvas()
    return getJpegBytes(canvas)
}

function blobToCanvas(blob, maxWidth, maxHeight) {
    return new Promise((resolve, reject) => {
        const img = new Image()
        img.onload = function () {
            const canvas = document.createElement('canvas')
            const scale = Math.min(
                1,
                maxWidth ? maxWidth / img.width : 1,
                maxHeight ? maxHeight / img.height : 1,
            )
            canvas.width = img.width * scale
            canvas.height = img.height * scale
            const ctx = canvas.getContext('2d')
            ctx.drawImage(img, 0, 0, img.width, img.height, 0, 0, canvas.width, canvas.height)
            resolve(canvas)
        }
        img.onerror = () => {
            reject(new Error('Error load blob to Image'))
        }
        img.src = URL.createObjectURL(blob)
    })
}

DEMO:

document.body.onclick = async () => {
    // take the screenshot
    var screenshotJpegBlob = await takeScreenshotJpegBlob()

    // show preview with max size 300 x 300 px
    var previewCanvas = await blobToCanvas(screenshotJpegBlob, 300, 300)
    previewCanvas.style.position = 'fixed'
    document.body.appendChild(previewCanvas)

    // send it to the server
    var formdata = new FormData()
    formdata.append("screenshot", screenshotJpegBlob)
    await fetch('https://your-web-site.com/', {
        method: 'POST',
        body: formdata,
        'Content-Type' : "multipart/form-data",
    })
}

// and click on the page

Solution 5

Here is a complete screenshot example that works with chrome in 2021. The end result is a blob ready to be transmitted. Flow is: request media > grab frame > draw to canvas > transfer to blob. If you want to do it more memory efficient explore OffscreenCanvas or possibly ImageBitmapRenderingContext

https://jsfiddle.net/v24hyd3q/1/

// Request media
navigator.mediaDevices.getDisplayMedia().then(stream => 
{
  // Grab frame from stream
  let track = stream.getVideoTracks()[0];
  let capture = new ImageCapture(track);
  capture.grabFrame().then(bitmap => 
  {
    // Stop sharing
    track.stop();
      
    // Draw the bitmap to canvas
    canvas.width = bitmap.width;
    canvas.height = bitmap.height;
    canvas.getContext('2d').drawImage(bitmap, 0, 0);
      
    // Grab blob from canvas
    canvas.toBlob(blob => {
        // Do things with blob here
        console.log('output blob:', blob);
    });
  });
})
.catch(e => console.log(e));

View more solutions

Comments

[Luke Stanley]

Very cool, Sikuli or Selenium might be good for going to different sites, comparing a shot of the site from the testing tool to your html2canvas.js rendered image in terms of pixel similarity! Wonder if you could automatically traverse parts of the DOM with a very simple formula solver to find how to parse alternate data sources for browsers where getBoundingClientRect isn't available. I'd probably use this if it was open source, was considering toying with it myself. Nice work Niklas!

[Niklas]

@Luke Stanley I will most likely throw the source up on github this weekend,still some minor clean ups and changes I want to do before then, as well as get rid of the unnecessary jQuery dependancy it currently has.

[Niklas]

Regarding getBoundingClientRect, the first version I actually did was without it, based on remembering parent element positions and applying padding's margins and all that manually, but the results weren't quite as accurate. With getBoundingClientRect you get pixel accurate results, even for text with a bit of manipulation on them.

[Luke Stanley]

cool stuff. About 46% of top sites use jQuery which means you can probably use a CDN'd version already in the browser cache. I wouldn't think it's a big deal to depend on it these days. source: trends.builtwith.com/javascript/JQuery

[Niklas]

The source code is now available at github.com/niklasvh/html2canvas, some examples of the script in use html2canvas.hertzen.com there. Still lot of bugs to fix, so I wouldn't recommend using the script in a live environment yet.

[Sergey Akopov]

There are drawing frameworks out there like Raphael which you could use here instead of HTML5 canvas, with the ability to make HTML5 canvas pluggable when it gains complete support from all major browsers. Just a thought. Great work.

[Mikhail]

Any intent on advancing this work with blackout feature, or configurable black out all $('.private') ?

[Niklas]

@Mikhail Sure, once the base screenshot capture is in a state where I feel like it could/should be used in production.

[Jagdeep]

any solution to make it work for SVG will be a great help. It does not work with highcharts.com

[Niklas]

@Jagdeep Its a bug in webkit

[Jagdeep]

@Niklas thanks for the response.. Is there any other alternate or any fix?

[Shivam Srivastava]

html2canvas.hertzen.com may helpful

[jwl]

i can't find any demos of just taking a screenshot -- everything is all about screen-sharing. will have to try it.

[Matt Sinclair]

@XMight, you can choose whether to allow this by toggling the screen capture support flag.

[Laurens Swart]

Does this still work? Because the example on your website gets stuck after submitting the text.

[StanE]

@XMight Please don't think like this. Web browsers should be able to do a lot of things, but unfortunately they are not consistent with their implementations. It is absolutely ok, if a browser has such functionality, as long as the user is being asked. No one will be able to make a screenshot without your attention. But too much fear results in bad implementations, like the clipboard API, which has been disabled altogether, instead creating confirmation dialogs, like for webcams, mics, screenshot capability, etc.

[1.21 gigawatts]

@StanE Precisely. The web notifications dialog is a good example of this. Ask the user permission on a site by site basis.

[Machado]

JS does not read the DOM, JS reads the document already rendered.

[Niklas]

@Machado, it reads the DOM and renders a representation of the document based on it. JS has no access to read what is being rendered by the browser.

[Machado]

TLDR: you use the DOM to access the document. "The Document Object Model provides a standard set of objects for representing HTML and XML documents, a standard model of how these objects can be combined, and a standard interface for accessing and manipulating them. [...]" (w3.org/TR/1998/REC-DOM-Level-1-19981001). You can't access the DOM, you can access the document itself, just because JS implements the DOM, which is a group of instructions.

[Niklas]

@Machado I have no problem accessing the DOM. For more information, a good introduction can be found at developer.mozilla.org/en-US/docs/Web/API/Document_Object_Mod‌​el/…

[idealizm]

@Niklas Congrats, great project. One (possibly stupid) question -- can it "screenshot" a Javascript widget that's called from an external server? Like, for example, a Facebook-provided "like us" widget? I noticed that the Twitter "follow us" widget didn't render. If not, is there a way?

[Niklas]

@idealizm It depends how the widget is implemented. If it adds a cross-origin iframe (like Twitter does), html2canvas doesn't have access to read the contents of that iframe. However, if the widget injects something into the dom of the current document, then html2canvas can render that fine.

[abzarak]

It is possible with Firefox now. Answer is partialy outdated

[Agustin Cautin]

This was deprecated and will be removed from the standard according to developer.mozilla.org/en-US/docs/Web/API/Navigator/getUserMe‌​dia

[zfrisch]

I think the reason why you're getting downvoted is most likely that html2canvas library is his library, not a tool that he simply pointed out.

[Jogai]

@Niklas I see your example grew into a real project. Maybe update your most upvoted comment about the experimental nature of the project. After almost 900 commits I would think its a bit more than an experiment at this point ;-)

[vintprox]

It is just fine if you don't want to capture post-processing effects (as blur filter).

[levant pied]

@AgustinCautin Navigator.getUserMedia() is deprecated, but just below it says "... Please use the newer navigator.mediaDevices.getUserMedia()", i.e. it was just replaced with a newer API.

[aravind3]

Limitations All the images that the script uses need to reside under the same origin for it to be able to read them without the assistance of a proxy. Similarly, if you have other canvas elements on the page, which have been tainted with cross-origin content, they will become dirty and no longer readable by html2canvas.

[Jay Dadhania]

Wonder why this had only 1 upvote, this proved to be really helpful!

[kabrice]

Please how does it works ? Can you provide a demo for newbies like me ? Thx

[Nikolay Makhonin]

@kabrice I added a demo. Just put the code in the Chrome console. If you need old browsers support, use: babeljs.io/en/repl

[Ginzorf]

It is a great piece of software - I'm curious if anyone has even been able to get it working with / updated to work with the Shadow DOM - multiple levels of Shadow DOM, for examples.

[Juan De la Cruz]

This took a picture of mysef in firefox... Just don't give permission to use the cam :)

[Nikolay Makhonin]

@juan-de-la-cruz I fixed "take cam picture" bug

[JavaDevSweden]

I liked this answer, but unfortunately it includes the "select screen to share" dialog in the screenshot, and that covers a large portion of the screen. At least for me using Chrome.

[jwh]

@JavaDevSweden only if you share the full screen, it looks like if you share a window or a browser tab the 'share' window isn't included.

[utopia]

Having CORS error problem on html2canvas. They seem to not find a solution as I'm writing this comment now. The attribute allowTaint and useCORS doesnt work