Optical Flow using OpenCV - Horizontal and Vertical Components

Question

I have the following code that finds the Optical Flow of 2 images (or 2 frames of a video) and it's colour coded. What I want is the horizontal and vertical components of the optical flow separately (as in separate images)

Here is the code I have so far:

import cv2
import numpy as np
frame1 = cv2.imread('my1.bmp')
frame2 = cv2.imread('my2.bmp')
prvs = cv2.cvtColor(frame1,cv2.COLOR_BGR2GRAY)
next = cv2.cvtColor(frame2,cv2.COLOR_BGR2GRAY)
hsv = np.zeros_like(frame1)
hsv[...,1] = 255

while(1):
    next = cv2.cvtColor(frame2,cv2.COLOR_BGR2GRAY)
    flow = cv2.calcOpticalFlowFarneback(prvs, next, 0.5, 3, 15, 3, 5, 1.2, 0)
    mag, ang = cv2.cartToPolar(flow[...,0], flow[...,1])
    hsv[...,0] = ang*180/np.pi/2
    hsv[...,2] = cv2.normalize(mag,None,0,255,cv2.NORM_MINMAX)
    rgb = cv2.cvtColor(hsv,cv2.COLOR_HSV2BGR)

    cv2.imshow('frame2',rgb)
    k = cv2.waitKey(30) & 0xff
    if k == 27:
        break
    elif k == ord('s'):
        cv2.imwrite('opticalmyhsv.pgm',rgb)

cap.release()
cv2.destroyAllWindows()

This is what the optical flow looks like given my two images:

Solution

If you want to visualize the horizontal and vertical component separately, you can visualize both separately as grayscale images. I'll make it such that a colour of gray denotes no motion, black denotes the maximum amount of motion in the frame going to the left (negative) while white denotes the maximum amount of motion in the frame going towards the right (positive).

The output of calcOpticalFlowFarneback is a 3D numpy array where the first slice denotes the amount of horizontal (x) displacement while the second slice denotes the amount of vertical (y) displacement.

As such, all you need to do is define two separate 2D numpy arrays that will store these values so we can display them to the user. However, you're going to need to normalize the flow for display such that no motion is a rough gray, motion to the extreme left is black, or intensity 0, and motion to the extreme right is white, or intensity 255.

Therefore, all you would need to do is modify your code to show two OpenCV windows for the horizontal and vertical motion like so:

import cv2
import numpy as np
frame1 = cv2.imread('my1.bmp')
frame2 = cv2.imread('my2.bmp')
prvs = cv2.cvtColor(frame1,cv2.COLOR_BGR2GRAY)
next = cv2.cvtColor(frame2,cv2.COLOR_BGR2GRAY)

flow = cv2.calcOpticalFlowFarneback(prvs, next, 0.5, 3, 15, 3, 5, 1.2, 0)

# Change here
horz = cv2.normalize(flow[...,0], None, 0, 255, cv2.NORM_MINMAX)     
vert = cv2.normalize(flow[...,1], None, 0, 255, cv2.NORM_MINMAX)
horz = horz.astype('uint8')
vert = vert.astype('uint8')

# Change here too
cv2.imshow('Horizontal Component', horz)
cv2.imshow('Vertical Component', vert)

k = cv2.waitKey(0) & 0xff
if k == ord('s'): # Change here
    cv2.imwrite('opticalflow_horz.pgm', horz)
    cv2.imwrite('opticalflow_vert.pgm', vert)

cv2.destroyAllWindows()

---

I've modified the code so that there is no while loop as you're only finding the optical flow between two predetermined frames. You're not grabbing frames off of a live source, like a camera, so we can just show both of the images not in a while loop. I've made the wait time for waitKey set to 0 so that you wait indefinitely until you push a key. This pretty much simulates your while loop behaviour from before, but it doesn't burden your CPU needlessly with wasted cycles. I've also removed some unnecessary variables, like the hsv variable as we aren't displaying both horizontal and vertical components colour coded. We also just compute the optical flow once.

In any case, with the above code we compute the optical flow, extract the horizontal and vertical components separately, normalize the components between the range of [0,255], cast to uint8 so that we can display the results then show the results. I've also modified your code so that if you wanted to save the components, it'll save the horizontal and vertical components as two separate images.

---

Edit

In your comments, you want to display a sequence of images using the same logic we have created above. You have a list of file names that you want to cycle through. That isn't very difficult to do. Simply take your strings and put them into a list and compute the optical flow between pairs of images by using the file names stored in this list. I'll modify the code such that when we reach the last element of the list, we will wait for the user to push something. Until then, we will cycle through each pair of images until the end. In other words:

import cv2
import numpy as np

# Create list of names here from my1.bmp up to my20.bmp
list_names = ['my' + str(i+1) + '.bmp' for i in range(20)]

# Read in the first frame
frame1 = cv2.imread(list_names[0])
prvs = cv2.cvtColor(frame1,cv2.COLOR_BGR2GRAY)

# Set counter to read the second frame at the start
counter = 1

# Until we reach the end of the list...
while counter < len(list_names):
    # Read the next frame in
    frame2 = cv2.imread(list_names[counter])
    next = cv2.cvtColor(frame2,cv2.COLOR_BGR2GRAY)

    # Calculate optical flow between the two frames
    flow = cv2.calcOpticalFlowFarneback(prvs, next, 0.5, 3, 15, 3, 5, 1.2, 0)

    # Normalize horizontal and vertical components
    horz = cv2.normalize(flow[...,0], None, 0, 255, cv2.NORM_MINMAX)     
    vert = cv2.normalize(flow[...,1], None, 0, 255, cv2.NORM_MINMAX)
    horz = horz.astype('uint8')
    vert = vert.astype('uint8')

    # Show the components as images
    cv2.imshow('Horizontal Component', horz)
    cv2.imshow('Vertical Component', vert)

    # Change - Make next frame previous frame
    prvs = next.copy()

    # If we get to the end of the list, simply wait indefinitely
    # for the user to push something
    if counter == len(list_names)-1
        k = cv2.waitKey(0) & 0xff
    else: # Else, wait for 1 second for a key
        k = cv2.waitKey(1000) & 0xff

    if k == 27:
        break
    elif k == ord('s'): # Change
        cv2.imwrite('opticalflow_horz' + str(counter) + '-' + str(counter+1) + '.pgm', horz)
        cv2.imwrite('opticalflow_vert' + str(counter) + '-' + str(counter+1) + '.pgm', vert)

    # Increment counter to go to next frame
    counter += 1

cv2.destroyAllWindows()

---

The above code will cycle through pairs of frames and wait for 1 second between each pair to give you the opportunity to either break out of the showing, or saving the horizontal and vertical components to file. Bear in mind that I have made it such that whatever frames you save, they are indexed with two numbers that tell you which pairs of frames they are showing. Before the next iteration happens, the next frame will be come the previous frame and so next gets replaced by a copy of prvs. At the beginning of the loop, the next frame gets read in appropriately.

---

Hope this helps. Good luck!

Comments

[a-Jays]

calcOpticalFlowPyrLK() is the Lucas-Kanade algorithm in OpenCV and it doesn't give you a two-element tuple of horizontal and vertical displacements. Just sayin'.

[rayryeng]

@a-Jays - Oops. My bad. In this case, it's a 3D numpy array. I obviously can't read the docs. Fixed my answer.

[rayryeng]

@Rohit - You can display all of them in a while loop like you did before. I'll post an edit.

[rayryeng]

I haven't had time to write an answer. It's Christmas after all! I will very soon.

[rayryeng]

@Rohit - Finished. Good luck!

[rayryeng]

@Rohit - It's possible, but the visualization won't make much sense. Optical flow is designed to capture the displacement between two frames. Showing overall optical flow for multiple frames would make interpretation both subjective and confusing. I don't really have an answer for you on how to show this in the best way possible.

[rayryeng]

@Rohit perhaps taking the components and averaging them may be an option? This way the horizontal and vertical components displayed show what the AVERAGE displacement would be over the sequence. Is this something you may want?

[rayryeng]

@Rohit no I haven't. Too much hassle for me to download. Can you just edit your post with the code?

[rayryeng]

@Rohit I may consider doing it later. On vacation. No promises though

[rayryeng]

@Rohit - stackoverflow.com/questions/6081008/…

[rayryeng]

You have to do it separately per component. a1 = flow[:,:,0]; numpy.savetxt("foo1.csv", a, delimiter=","); a2 = flow[:,:,1]; numpy.savetxt("foo2.csv", a, delimiter=",");. I won't comment on this question anymore. I consider this solved. Good luck!