update frame in matplotlib with live camera preview

Question

I am new to both Python and Matplotlib. My computer is connected to two usb cameras, and I intend to use the subplot(1,2,1) and subplot(1,2,2) in matplotlib to plot the frames from the two camera in time series. When I do this with my code, I either get only one frame plotted or get a black screen in the plotting area.

My code look like below

#import
import cv2
import matplotlib.pyplot as plt

#Initiate the two cameras
cap1 = cv2.VideoCapture(0)
cap2 = cv2.VideoCapture(1)

#Capture the frames from camera 1 and 2 and display them over time using matplotlib

while True:
    #grab frame from camera 1 and 2
    ret1,frame1 = cap1.read()
    ret2,frame2 = cap2.read()

    plt.subplot(1,2,1), plt.imshow(cv2.cvtColor(frame1,cv2.COLOR_BGR2RGB))
    plt.subplot(1,2,2), plt.imshow(cv2.cvtColor(frame2,cv2.COLOR_BGR2RGB))

    #draw the plot
    plt.show(False)
    #Result is black screen. If plt.show() is called, I see the frames but then it freezes.

Solution

Interactive mode

One way of updating a plot in matplotlib is to use interactive mode (plt.ion()). You should then not recreate new subplots for each frame you capture, but create your plot with images once and update it afterwards.

import cv2
import matplotlib.pyplot as plt

def grab_frame(cap):
    ret,frame = cap.read()
    return cv2.cvtColor(frame,cv2.COLOR_BGR2RGB)

#Initiate the two cameras
cap1 = cv2.VideoCapture(0)
cap2 = cv2.VideoCapture(1)

#create two subplots
ax1 = plt.subplot(1,2,1)
ax2 = plt.subplot(1,2,2)

#create two image plots
im1 = ax1.imshow(grab_frame(cap1))
im2 = ax2.imshow(grab_frame(cap2))

plt.ion()

while True:
    im1.set_data(grab_frame(cap1))
    im2.set_data(grab_frame(cap2))
    plt.pause(0.2)

plt.ioff() # due to infinite loop, this gets never called.
plt.show()

FuncAnimation

Another option is of course to use matplotlib's built in FuncAnimation which is especially designed to animate plots.

import cv2
import matplotlib.pyplot as plt
from matplotlib.animation import FuncAnimation

def grab_frame(cap):
    ret,frame = cap.read()
    return cv2.cvtColor(frame,cv2.COLOR_BGR2RGB)

#Initiate the two cameras
cap1 = cv2.VideoCapture(0)
cap2 = cv2.VideoCapture(1)

#create two subplots
ax1 = plt.subplot(1,2,1)
ax2 = plt.subplot(1,2,2)

#create two image plots
im1 = ax1.imshow(grab_frame(cap1))
im2 = ax2.imshow(grab_frame(cap2))

def update(i):
    im1.set_data(grab_frame(cap1))
    im2.set_data(grab_frame(cap2))
    
ani = FuncAnimation(plt.gcf(), update, interval=200)
plt.show()

In order to close the window on a key press event, you may add a callback, like so

#... other code
ani = FuncAnimation(plt.gcf(), update, interval=200)

def close(event):
    if event.key == 'q':
        plt.close(event.canvas.figure)

cid = plt.gcf().canvas.mpl_connect("key_press_event", close)

plt.show()

# code that should be executed after window is closed.

Comments

[jxw]

This one really helps. Are there any possibilities to allow the user to exit the while True:... loop by pressing a key without the use of CTRL + C key? This is to allow execution of the next lines in the script. I have tried the cv2.waitkey, but it doesn't seem to help

[ImportanceOfBeingErnest]

What exactly do you want to happen? So user presses Key 'Q' and then? Shoud the window close or should it stay open? In which environment are you running this? As a script?

[jxw]

I am using Python to run this in Windows computer. When user press the 'q' key, the window should be closed and exit the while loop. Right now I have modified my code to while True: frame1 = grab_frame(cap1) im1.set_data(frame1) im2.set_data(frame1) plt.pause(0.1) k = cv2.waitKey(1) & 0xFF if k==ord('q'): break but when running the code nothing happens when I type q in my keyboard.

[ImportanceOfBeingErnest]

cv2.waitKey(1) is an opencv command, it will not work when using matplotlib. Is there any reason you want to press a key instead of hitting the X of the window to close it?

[jxw]

I was not aware that cv2.waitKey will not work with matplotlib. It is fine hitting the X and close the window, but I want to get out of the while loop and continue with my python script. Right now when hitting the X, one also stops the script from running the subsequent lines outside the while loop. Anyways to continue to run the subsequent lines outside the while loop?

[ImportanceOfBeingErnest]

Yep, you should not use the while loop at all and take the second solution. I updated the answer to include the key press, but if you're happy with using the X button, you may leave it as it is and just add some code after plt.show().