Match 3D point cloud to CAD model

Question

I have a point cloud of an object, obtained with a laser scanner, and a CAD surface model of that object.

How can I match the point cloud to the surface, to obtain the translation and rotation between cloud and model?

I suppose I could sample the surface and try the Iterative Closest Point (ICP) algorithm to match the resulting sampled point cloud to the scanner point cloud.

Would that actually work?

And are there better algorithms for this task?

Solution 1

Yes, ICP can be applied to this problem, as you suggest with sampling the surface. It would be best if you have all available faces in your laser scan otherwise you may have to remove invisible faces from your model (depending on how many of these there are).

One way of automatically preparing a model by getting rid of some of the hidden faces is to calculate the concave hull which can be used to discard hidden faces (which are for example faces that are not close to the concave hull). Depending on how involved the model is this may or may not be necessary.

ICP works well if given a good initial guess because it ignores points that are not close with respect to the current guess. If ICP is not coming up with a good alignment you may try it with multiple random restarts to try and fix this problem, choosing the best alignment.

A more involved solution is to do local feature matching. You sample and calculate an invariant descriptor like SHOT or FPFH. You find the best matches, reject non-consistent matches, use them to come up with a good initial alignment and then refine with ICP. But you may not need this step depending on how robust and fast the random-restart ICP is.

Solution 2

In new OpenCV, I have implemented a surface matching module to match a 3D model to a 3D scene. No initial pose is required and the detection process is fully automatic. The model also involves an ICP.

To get an idea, please check that out a video here (though it is not generated by the implementation in OpenCV):

https://www.youtube.com/watch?v=uFnqLFznuZU

The full source code is here and the documentation is here.

You mentioned that you needed to sample your CAD model. This is correct and we have given a sampling algorithm suited for point pair feature matching, such as the one implemented in OpenCV:

Birdal, Tolga, and Slobodan Ilic. A point sampling algorithm for 3D matching of irregular geometries. 2017 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). IEEE, 2017.

http://campar.in.tum.de/pub/tbirdal2017iros/tbirdal2017iros.pdf

Solution 3

There's an open source library for point cloud algorithms which implements registration against other point clouds. May be you can try some of their methods to see if any fit.

As a starter, if they don't have anything specific to fit against a polygon mesh, you can treat the mesh vertices as another point cloud and fit your point cloud against it. This is something that they definitely support.

Comments

[HugoRune]

The video looks interesting, but this answer lacks the required information to reproduce your results. Could you summarize the approach you used, and show the relevant OpenCv calls? Is the full code available somewhere?

[Tolga Birdal]

Full code is available at github.com/tolgabirdal/opencv_contrib/tree/master/modules/…

[Schütze]

Is there a tutorial on this? How to train, how to use and etc?

[Schütze]

You said the demonstrated video wasn't taken using the code. Also, it's not clear how to train and test, there is no so to speak a usage guideline. The link you shared "describes" the method. People would first try it out if it works at all for their case, then would try delving into the description.

[Tolga Birdal]

Well if you scroll down, you'll see the sample code that trains and tests it. The section is called "A Complete Sample". The other code (for the video) is not yet open source. So there is no tutorial that I can provide.

[bob]

Hi. I would like to know if is this code can solve the problem when the objects are one on top of the other? So when objects are identical and one on top of others but with different orientation , and I have to get the orientation and the position of the object on the top. Would it be working in that case?

[Tolga Birdal]

In theory yes. Doing that is also easy, just take multiple hypotheses output and look at the vector. In practice, you might get some reduced performance as the accumulators might get mystified. If you like to make sure that things go right: (1) detect the object with highest confidence, (2) remove points found to be on the detection from the scene, (3) run the detection again. This makes things slower of course.

[Macedon971]

Hi. I would like to use surface matching for real-time 6D Pose estimation and tracking the object in the scene, Is it possible to do that with the sample code you provided in the tutorial? First I tried to run the code with customs data but never was able to get some results. So what are the requirements for the object and scene file? The scene file must be segmented and downsample and only contaion objects (no planar surfaces like table or floor or vertical surfaces like wall). Please can help?

[Tolga Birdal]

Make sure the data is in some metric scale (no scale is estimated), the normals are computed reasonably well and correctly aligned (I don't handle the sign ambiguity, so flip towards the camera). Also make sure that the sampling is between 0.6-0.95. Feel free to shoot me an e-mail in case of further questions.