How can I detect an object in image frame using OpenCV?

Question

I am developing a rover using Raspberry Pi , that will sweep a room and pick up objects fallen on the ground . To detect the object , I am using a reference Image , that is taken right at the start of the rover's operation , and an Image (new Image) that is clicked every 10 seconds . To determine if there is a change in the image frame , I do a image subtraction between the reference image and the new image . If any difference is found , it'll draw a contour around it , and if the contour area is greater than a certain threshold (cautionary step) , it concludes that there exists an object .

I am using the following code -

import numpy as np
import cv2,time

img=cv2.imread("object1.jpg")
img1=cv2.imread("object2.jpg")
sub=cv2.subtract(img,img1)

gray=cv2.cvtColor(sub,cv2.COLOR_BGR2GRAY)
blur = cv2.GaussianBlur(gray,(3,3),0)
_, contours, _= cv2.findContours(blur,cv2.RETR_TREE,cv2.CHAIN_APPROX_SIMPLE)
c=max(contours,key=cv2.contourArea)
print(cv2.contourArea(c))

if cv2.contourArea>20000:
   print("Object detected !")

The above code just uses 2 images to calculate their difference and determine if there is an object present .Please note that I have not posted the original code here that I will be using in my project .

Now , the above code works fine for very controlled situations, say ,when the image background is very constant or there is no presence of shadow in it . But considering the fact that the rover will be moving around in the room , and there are chances that the lighting variations might trigger a false object detection , even if there is no real object in the frame . The difference might be triggered due to false contouring from shadow effects .

I want to know , if there is any other way of achieving this object detection without doing foreground/background image subtraction . I have also considered using a ultrasonic sensor to detect the object's presence , however that is not a very reliable option . I would prefer a Image processing based solution .

Thank you .

==========================================================================

EDIT 1 -

So , I decided to change the algorithm a bit . I have done thresholding on both the foreground and background Images and then performed absdiff between the binary images , to obtain any frame change(object) . The code is as follows -

import numpy as np
import cv2,time

img1=cv2.imread("back.jpeg")
blur1 = cv2.GaussianBlur(img1,(5,5),0)
gray1=cv2.cvtColor(blur1,cv2.COLOR_BGR2GRAY)
ret,thresh1 = cv2.threshold(gray1,65,255,cv2.THRESH_BINARY_INV)

img2=cv2.imread("front.jpeg")
blur2 = cv2.GaussianBlur(img2,(5,5),0)
gray2=cv2.cvtColor(blur2,cv2.COLOR_BGR2GRAY)
ret,thresh2 = cv2.threshold(gray2,65,255,cv2.THRESH_BINARY_INV)

diff=cv2.absdiff(thresh2,thresh1)
diff=cv2.bitwise_xor(diff,thresh1)

kernel = np.ones((2,2),np.uint8)
diff=cv2.erode(diff,kernel,iterations = 1)
diff=cv2.dilate(diff,kernel,iterations = 8)

_, contours, _= cv2.findContours(diff,cv2.RETR_TREE,cv2.CHAIN_APPROX_SIMPLE)
c=max(contours,key=cv2.contourArea)
x,y,w,h = cv2.boundingRect(c)
cv2.rectangle(diff,(x,y),(x+w,y+h),(125,125,125),2)


cv2.imshow("thresh",diff)
cv2.waitKey(0)

"absdiff" is followed by Erosion and Dilation . After that , I find the largest contour and determine if there is an object . The images used in the algorithm are as follows -

1. Background Image - Background Image

2. Foreground Image - Foreground image

3. Foreground Threshold - Foreground threshold Image

4. Background Threshold - Background threshold Image

5. Difference Image - Final Image with contour and its boundary .

As you can see , the detection works fine . I have few other foreground images which I have used to test the algorithm . They have given satisfactory results .I wanted to know , if there is any other way of achieving the same result with better efficiency .

PS- All of the foreground Images have been taken with Flash ON . I have tried with Flash OFF but there seems to exist a lot of noise in the image .

=============================================================

EDIT 2-

Performance of the algo using other pictures -

Note :- The Background Image has remained same .

1. Object 1 - Foreground Image 1
2. Object 1 Detection - Foreground Image 1 Result

Solution

I doubt if this problem is that much simple as you have described in the question, it will be get very complex when we move in real world scenario.

But anyway assuming you have small objects only present in the room then you identify them by identifying connected components in the binary image captured and choose them based on their relative pixels sizes.

Here is Python implementation for the same:

img = cv2.imread('D:/Image/objects.jpg')
gray = cv2.cvtColor(img, cv2.COLOR_BGR2GRAY)
# binarize the image
ret, bw = cv2.threshold(gray, 128, 255, 
cv2.THRESH_BINARY_INV+cv2.THRESH_OTSU)

# find connected components
connectivity = 4
nb_components, output, stats, centroids = 
cv2.connectedComponentsWithStats(bw, connectivity, cv2.CV_32S)
sizes = stats[1:, -1]; nb_components = nb_components - 1
min_size = 250 #threshhold value for objects in scene
img2 = np.zeros((img.shape), np.uint8)
for i in range(0, nb_components+1):
    # use if sizes[i] >= min_size: to identify your objects
    color = np.random.randint(255,size=3)
    # draw the bounding rectangele around each object
    cv2.rectangle(img2, (stats[i][0],stats[i][1]),(stats[i][0]+stats[i][2],stats[i][1]+stats[i][3]), (0,255,0), 2)
    img2[output == i + 1] = color

Image containing objects:

Detected objects using connected components labels:

Comments

[Micka]

with object template images you can use keypoint matching like sift/surf/orb but you should have a template at least from each side of the object.

[Boudhayan Dev]

@Micka Well the object is going to differ from time to time . It can be a bottle, plastic or anything else . If your solution covers the case I mentioned , then please elaborate . I am really not aware of sift/orb method .

[Micka]

you must provide templates for each of the objects. In one of the original sift papers there is a description on how to use keypoint matching for object detection. But if there isnt enough texture (transparent bottle without label?) keypoint matching will probably not work.

[Boudhayan Dev]

@Micka Please check the Edit.

[Boudhayan Dev]

The objects in my image are not going to be entirely black or white . I know I can do a gray image conversion but it will still not result into a complete binary image . I will need to do a threshold to achieve a complete binary image. And If I do thresholding , wouldn't it be better to just find contours and find the contour with a certain size , to detect and object ? What is the use of connected components here ?

[Boudhayan Dev]

Please check the EDIT. @flamelite

[flamelite]

ok you captured those images by yourself but how would you make your rover to get the foreground and background image of the same place? Image subtraction would work for the image frames of the same position only.

[Boudhayan Dev]

Yes I am aware of that shortcoming . But the working assumption is that the background Image is not going to change abruptly . As you can see in my background and foreground Images , the background ,i.e the white tile is almost constant , except probably the tile edges . I know frame subtraction will not give a completely clean result . But it will be enough to detect the foreground object . Also , the background Image will be clicked on rover startup and it will be swapped and replaced every 10 sec with the foreground image (if object is not present) .

[flamelite]

i am not sure about that, may be you should prepare a prototype based on both the ideas and test which one works then you would get more short comings of the algorithm and change/modify accordingly.

[flamelite]

@JohnOliver did you make any progress?

[Boudhayan Dev]

I am sticking with my approach for now . I have used few more foreground objects and tested the algorithm , it seems to work . I'll attach those pictures and their results in the EDIT section in a while .

[flamelite]

sure thanks, after thinking on your problem now i am also exited to make a simple device like yours and i am also doing some research :)

[Boudhayan Dev]

Cool. Update here , incase you figure out a better solution. All the best !