Using tesseract to recognize license plates

Question

I'm developing an app which can recognize license plates (ANPR). The first step is to extract the licenses plates from the image. I am using OpenCV to detect the plates based on width/height ratio and this works pretty well:

But as you can see, the OCR results are pretty bad.

I am using tesseract in my Objective C (iOS) environment. These are my init variables when starting the engine:

// init the tesseract engine.
    tesseract = new tesseract::TessBaseAPI();
    int initRet=tesseract->Init([dataPath cStringUsingEncoding:NSUTF8StringEncoding], [language UTF8String]);
    tesseract->SetVariable("tessedit_char_whitelist", "BCDFGHJKLMNPQRSTVWXYZ0123456789-");
    tesseract->SetVariable("language_model_penalty_non_freq_dict_word", "1");
    tesseract->SetVariable("language_model_penalty_non_dict_word ", "1");
    tesseract->SetVariable("load_system_dawg", "0");

How can I improve the results? Do I need to let OpenCV do more image manipulation? Or is there something I can improve with tesseract?

Solution 1

Two things will fix this completely:

1. Remove everything which is not text from the image. You need to use some CV to find the plate area (for example by color, etc) and then mask out all of the background. You want the input to tesseract to be black and white, where text is black and everything else is white

2. Remove skew (as mentioned by FrankPI above). tesseract is actually supposed to work okay with skew (see "Tesseract OCR Engine" overview by R. Smith) but on the other hand it doesn't always work, especially if you have a single line as opposed to a few paragraphs. So removing skew manually first is always good, if you can do it reliably. You will probably know the exact shape of the bounding trapezoid of the plate from step 1, so this should not be too hard. In the process of removing skew, you can also remove perspective: all license plates (usually) have the same font, and if you scale them to the same (perspective-free) shape the letter shapes would be exactly the same, that would help text recognition.

Some further pointers...

Don't try to code this at first: take a really easy to OCR (ie: from directly in front, no perspective) picture of a plate, edit it in photoshop (or gimp) and run it through tesseract on the commandline. Keep editing in different ways until this works. For example: select by color (or flood select the letter shapes), fill with black, invert selection, fill with white, perspective transform so corners of plate are a rectangle, etc. Take a bunch of pictures, some harder (maybe from odd angles, etc). Do this with all of them. Once this works completely, think about how to make a CV algorithm that does the same thing you did in photoshop :)

P.S. Also, it is better to start with higher resolution image if possible. It looks like the text in your example is around 14 pixels tall. tesseract works pretty well with 12 point text at 300 dpi, this is about 50 pixels tall, and it works much better at 600 dpi. Try to make your letter size be at least 50 preferably 100 pixels.

P.P.S. Are you doing anything to train tesseract? I think you have to do that, the font here is different enough to be a problem. You probably also need something to recognize (and not penalize) dashes which will be very common in your texts, looks like in the second example "T-" is recognized as H.

Solution 2

I don't know tesseract too much, but I have some information about OCR. Here we go.

• In an OCR task you need to be sure that, your train data has the same font that you are trying to recognize. Or if you are trying to recognize multiple fonts, be sure that you have those fonts in your train data to get best performance.
• As far as I know, tesseract applies OCR in few different ways: One, you give an image which has multiple letters in it and let tesseract do the segmentation. And other, you give segmented letters to tesseract and only expect it to recognize the letter. Maybe you can try to change the one which you are using.
• If you are training recognizer by yourself be sure that you have enough and equally amount of each letter in your train data.

Hope this helps.

Solution 3

I've been working on an iOS app, if you need to improve the results you should train tesseract OCR, this improved 90% for me. Before tranning, OCR results were pretty bad.

So, I used this gist in the past to train tesseract ORC with a licence plate font.

If you are interested, I open-sourced this project some weeks ago on github

Solution 4

Now License Plate can be easily recognized by mlmodel. I have created the core model you can find it here . You just need to split characters in 28*28 resolution through vision framework and send this image to VNImageRequestHandler like given below-

let handler = VNImageRequestHandler(cgImage: imageUI.cgImage!, options: [:])

you will get desired results by using my core mlmodel. Use this link for better clarification but use my model for better results in license plate recognition. I have also created the mlmodel for License Plate Recognition.

Solution 5

Here is my real world example with trying out OCR from my old power meter. I would like to use your OpenCV code so that OpenCV does automatic cropping of image, and I'll do image cleaning scripts.

• First image is original image (croped power meter numbers)
• Second image is slightly cleaned up image in GIMP, around 50% OCR accuracy in tesseract
• Third image is completely cleaned image - 100% OCR recognized without any training!

View more solutions

Comments

[sansuiso]

Ad hoc training is clearly a good suggestion, it can even "learn" a part of the image defaults (skew...).

[unicorn80]

Thanks. Already did step 1 and 2 (although not showing in my example). I used this tutorial: felix.abecassis.me/2011/10/opencv-rotation-deskewing I trained tesseract with the font but I doesn't get any better.

[Alex I]

@inversivemedia: Could you post a few examples of the images as they are just before you feed them to tesseract? Also, what results do you get from command line tesseract? (and what options did you use to run it?). I have got excellent results (99.9% accuracy on characters) from trained tesseract on a large body of text in the same font, so I am surprised you are having problems with it.

[Christian Ternus]

Great, comprehensive answer.

[unicorn80]

Did you clean the image using GIMP? I need a way to clean the image using the mobile device (opencv/ios)

[valentt]

This cleanup was done manually in GIMP, this was fastest way to see how good tesseract can be if I get clean image for it to process. Now there is much more legwork to make image clean with imagemagic and scripts. I'll also try Scan Tailor package I see has some great options for cleaning images.

[Parth Doshi]

@valentt: can you share the code for cleaning the image using opencv?

[valentt]

@ParthDoshi image cleanup was done manually just for feasibility test... and we didn't go further with the project.

[Istiaque Ahmed]

@AlexI, may I ask you to have a look at a Tesseract and OpenCV related question here : github.com/yardstick17 ?

[Istiaque Ahmed]

May I ask you to have a look at a Tesseract and OpenCV related question here : stackoverflow.com/questions/66946835/… ?