Is it possible to display an OpenCV video inside the IPython /JuPyter Notebook?

Question

When running the examples from the OpenCV video processing python tutorials, they all pop up in a dedicated window. I know that the IPython notebook can display videos from disk and YouTube, so I wonder if there is a way to direct the OpenCV video playback to the Notebook browser and have it play in the output cell instead of a separate window (preferably without saving it to disk and then playing it from there).

Below is the code from the OpenCV tutorial.

import cv2

cap = cv2.VideoCapture('/path/to/video') 

while(True):
    # Capture frame-by-frame
    ret, frame = cap.read()

    # Our operations on the frame come here
    gray = cv2.cvtColor(frame, cv2.COLOR_BGR2GRAY)

    # Display the resulting frame
    cv2.imshow('frame',gray)
    if cv2.waitKey(1) & 0xFF == ord('q'):
        break

# When everything done, release the capture
cap.release()
cv2.destroyAllWindows()

Solution 1

You can do it with Bokeh and probably it is a bit faster.

from bokeh.plotting import figure
from bokeh.io import output_notebook, show, push_notebook
import cv2
import time
output_notebook()

cap = cv2.VideoCapture(0)
ret, frame = cap.read()
frame=cv2.cvtColor(frame, cv2.COLOR_BGR2RGBA) # because Bokeh expects a RGBA image
frame=cv2.flip(frame, -1) # because Bokeh flips vertically
width=frame.shape[1]
height=frame.shape[0]
p = figure(x_range=(0,width), y_range=(0,height), output_backend="webgl", width=width, height=height)
myImage = p.image_rgba(image=[frame], x=0, y=0, dw=width, dh=height)
show(p, notebook_handle=True)
while True:
    ret, frame = cap.read()
    frame=cv2.cvtColor(frame, cv2.COLOR_BGR2RGBA)
    frame=cv2.flip(frame, -1)
    myImage.data_source.data['image']=[frame]
    push_notebook()
    time.sleep(0.3)

Solution 2

To make the display faster use only IPython.display.display inside the notebook and JPG format instead of PNG. (Note displaying with cv2.imshow natively outside the notebook is much faster, but this is not what the question asks for):

The code below will test all the supported file formats to find the fastest one (extracted from __doc__ with a regex, not reliable)

from IPython.display import clear_output, Image, display, HTML
import cv2

# Read one frame from the camera for testing
video = cv2.VideoCapture(0)
_, frame = video.read()
video.release()

import re
from timeit import timeit
import math

extensions=re.findall(r"\\\*(\.\w*)", cv2.imread.__doc__)

def test(extension):
    try:
        totalTime=0
        numTry=3
        for _ in range(numTry):
            totalTime+=timeit(lambda: display(Image(data=cv2.imencode(extension, frame)[1])), number=1)
            clear_output(wait=True)
        return totalTime/numTry, extension

    except cv2.error as e: #usually "unsupported file type"
        return (math.inf, extension, e)
for x in sorted(
    [test(extension) for extension in extensions], key=lambda x: x[0]
): print(x)

In my case, .jpeg is the fastest. Make sure that the browser display also support that extension:

Image(data=cv2.imencode(".jpeg", frame)[1].tobytes())

Then, to play the video:

import cv2
from IPython.display import display, Image

video = cv2.VideoCapture(0)
display_handle=display(None, display_id=True)
try:
    while True:
        _, frame = video.read()
        frame = cv2.flip(frame, 1) # if your camera reverses your image
        _, frame = cv2.imencode('.jpeg', frame)
        display_handle.update(Image(data=frame.tobytes()))
except KeyboardInterrupt:
    pass
finally:
    video.release()
    display_handle.update(None)

update is a little faster than clear_output + display every time; however compare to the rendering it isn't a significant improvement.

Solution 3

The video encoded data (if in a format the browser can decode, eg. h264-encoded in ISO mp4 container) can be displayed using an HTML <video> tag and IPython.core.display.HTML(), this will provide standard playback performance.

The <video> can be a link, or have embedded base64'ed data (the latter is what matplotlib.animation does, for example), and its data can of course be generated in your notebook, using OpenCV (eg. VideoWriter).

Solution 4

Yes. But it will be slooowwww....

Code with Python 3 and OpenCV 3.3 that reads from webcam (from file, just change cv2.VideoCapture("filename.mp4")):

from IPython.display import clear_output, Image, display, HTML
import numpy as np
import cv2
import base64

def arrayShow (imageArray):
    ret, png = cv2.imencode('.png', imageArray)
    encoded = base64.b64encode(png)
    return Image(data=encoded.decode('ascii'))
video = cv2.VideoCapture(0)
while(True):
    try:
        clear_output(wait=True)
        _, frame = video.read()
        lines, columns, _ =  frame.shape
        frame = cv2.resize(frame, (int(columns/4), int(lines/4))) 
        img = arrayShow(frame)
        display(img)
    except KeyboardInterrupt:
        video.release()

You may need to change IOPub data rate limit. You can change this in your .jupyter config or just run jupyter notebook --NotebookApp.iopub_data_rate_limit=1000000000

The keyboard interrupt doesn't work properly, though.

View more solutions

Comments

[joelostblom]

Thanks for this suggestion. Unfortunately it is not working for me. I tried with a file from disk, but the video lags significantly and only advances 1 frame every few seconds (I tried both JupyterNotebook and JupyterLab). Is playback as smooth for you as when playing the video in a dedicated player such as VLC?

[diimdeep]

Works well! You can interrupt easily from menu.

[stories2]

In python3 you should change Image(data=encoded.decode('ascii')) to Image(data=png)

[user202729]

You need to add break in the except block.

[user202729]

Note that recent version of bokeh requires (w, h) array of uint32, while opencv outputs (w, h, 4) array of uint8. So it's necessary to convert between them. (use frame = frame.view(dtype=numpy.uint8).reshape(frame.shape[:2]) for example) Although I still find this method a little slower than use IPython.display with jpeg format (see my answer), and much slower than cv2.imshow in a separate window.

[user202729]

Remark: at the time of testing this is faster than Bokeh. Besides you can do Image(data=<numpy array>) as well, although that usage isn't explicitly documented