How to remove a "green screen" portrait background

Question

I'm looking for a way to automatically remove (=make transparent) a "green screen" portrait background from a lot of pictures.

My own attempts this far have been... ehum... less successful.

I'm looking around for any hints or solutions or papers on the subject. Commercial solutions are just fine, too.

And before you comment and say that it is impossible to do this automatically: no it isn't. There actually exists a company which offers exactly this service, and if I fail to come up with a different solution we're going to use them. The problem is that they guard their algorithm with their lives, and therefore won't sell/license their software. Instead we have to FTP all pictures to them where the processing is done and then we FTP the result back home. (And no, they don't have an underpaid staff hidden away in the Philippines which handles this manually, since we're talking several thousand pictures a day...) However, this approach limits its usefulness for several reasons. So I'd really like a solution where this could be done instantly while being offline from the internet.

EDIT: My "portraits" depictures persons, which do have hair - which is a really tricky part since the green background will bleed into hair. Another tricky part is if it is possible to distingush between the green in the background and the same green in peoples clothes. The company I'm talking about above claims that they can do it by figuring out if the green area are in focus (being sharp vs blurred).

Solution 1

Since you didn't provide any image, I selected one from the web having a chroma key with different shades of green and a significant amount of noise due to JPEG compression.

There is no technology specification so I used Java and Marvin Framework.

input image:

The step 1 simply converts green pixels to transparency. Basically it uses a filtering rule in the HSV color space.

As you mentioned, the hair and some boundary pixels presents colors mixed with green. To reduce this problem, in the step 2, these pixels are filtered and balanced to reduce its green proportion.

before:

after:

Finally, in the step 3, a gradient transparency is applied to all boundary pixels. The result will be even better with high quality images.

final output:

Source code:

import static marvin.MarvinPluginCollection.*;

public class ChromaToTransparency {

    public ChromaToTransparency(){
        MarvinImage image = MarvinImageIO.loadImage("./res/person_chroma.jpg");
        MarvinImage imageOut = new MarvinImage(image.getWidth(), image.getHeight());
        // 1. Convert green to transparency
        greenToTransparency(image, imageOut);
        MarvinImageIO.saveImage(imageOut, "./res/person_chroma_out1.png");
        // 2. Reduce remaining green pixels
        reduceGreen(imageOut);
        MarvinImageIO.saveImage(imageOut, "./res/person_chroma_out2.png");
        // 3. Apply alpha to the boundary
        alphaBoundary(imageOut, 6);
        MarvinImageIO.saveImage(imageOut, "./res/person_chroma_out3.png");

    }

    private void greenToTransparency(MarvinImage imageIn, MarvinImage imageOut){
        for(int y=0; y<imageIn.getHeight(); y++){
            for(int x=0; x<imageIn.getWidth(); x++){

                int color = imageIn.getIntColor(x, y);
                int r = imageIn.getIntComponent0(x, y);
                int g = imageIn.getIntComponent1(x, y);
                int b = imageIn.getIntComponent2(x, y);

                double[] hsv = MarvinColorModelConverter.rgbToHsv(new int[]{color});

                if(hsv[0] >= 60 && hsv[0] <= 130 && hsv[1] >= 0.4 && hsv[2] >= 0.3){
                    imageOut.setIntColor(x, y, 0, 127, 127, 127);
                }
                else{
                    imageOut.setIntColor(x, y, color);
                }

            }
        }
    }

    private void reduceGreen(MarvinImage image){
        for(int y=0; y<image.getHeight(); y++){
            for(int x=0; x<image.getWidth(); x++){
                int r = image.getIntComponent0(x, y);
                int g = image.getIntComponent1(x, y);
                int b = image.getIntComponent2(x, y);
                int color = image.getIntColor(x, y);
                double[] hsv = MarvinColorModelConverter.rgbToHsv(new int[]{color});

                if(hsv[0] >= 60 && hsv[0] <= 130 && hsv[1] >= 0.15 && hsv[2] > 0.15){
                    if((r*b) !=0 && (g*g) / (r*b) >= 1.5){
                        image.setIntColor(x, y, 255, (int)(r*1.4), (int)g, (int)(b*1.4));
                    } else{
                        image.setIntColor(x, y, 255, (int)(r*1.2), g, (int)(b*1.2));
                    }
                }
            }
        }
    }

    public static void main(String[] args) {
        new ChromaToTransparency();
    }
}

Solution 2

If you know the "green color" you may write a small program in opencv C/C++/Python to do extract that color and replace with transparent pixels.

Solution 3

123 Video Magic Green Screen Background Software and there are a few more just made to remove green screen background hope this helps

Solution 4

Take a look at this thread: http://www.wizards-toolkit.org/discourse-server/viewtopic.php?f=2&t=14394&start=0

and the link within it to the tutorial at: http://tech.natemurray.com/2007/12/convert-white-to-transparent.html

Then it's just a matter of writing some scripts to look through the directory full of images. Pretty simple.

Solution 5

PaintShop Pro allows you to remove backgrounds based on picking a color. They also have a Remove Background wand that will remove whatever you touch (converting those pixels to transparent). You can tweak the "tolerance" for the wand, such that it takes out pixels that are similar to the ones you are touching. This has worked pretty well for me in the past.

To automate it, you'd program a script in PSP that does what you want and then call it from your program. This might be a kludgy way to to do automatic replacement, but it would be the cheapest, fastest solution without having to write a bunch of C#/C++ imaging code or pay a commercial agency.

They being said, you pay for what you get.

View more solutions

Comments

[k00k]

I believe he's looking for an automated "server-side" solution.

[Dan Byström]

First I said automatically. And this cannot possibly solve the problem with the green color being reflected and shining through (especially blonde) hair. That requires quite a bit of manual work. And how many peoble would it take to do this to 5000 images a day...?

[sohtimsso1970]

PSP can be automated. It might be kludgy but you can write scripts that run by command.

[k00k]

I should note, I've not tried this, and YMMV. Especially when it comes to anti-aliasing and soft edges.

[Dan Byström]

I can find the green color by inspecting the areas which I know should contain mostly green and calculating the median value... but the problem is hair, hair, hair... the green color bleeds into the hair and mixes with it... and then we have the problem with clothes in the same green color...

[rics]

Based on your description above, my answer is correct. Based on you comment it is not. But why should I know it in advance to merit the downvote? Happy searching.

[Dan Byström]

Yes, I think that too. I wasn't the one who downvoted you. I consider your answer fully valid (although not appicable - as noted).

[sohtimsso1970]

You've mentioned the hair issue several times (here and in an answer below). That is why I suggested programming against a commercial product like PSP. It already contains algorithms to figure out what pixels are "similar shades of green" and which are not. You don't want to try and do that stuff yourself. Find a commercial product (like PSP) that offers scripting and shell it open.

[paxdiablo]

upvoted since I think the downvote was unfair, regardless of the source. I misunderstood the question so it's likely that lesser mortals may have as well :-) You may want to consider deleting the answer while you're ahead if you agree it doesn't match the question.

[paxdiablo]

upvoted since I think the downvote was unfair, regardless of the source. I misunderstood the question so it's likely that lesser mortals may have as well :-) You may want to consider deleting the answer while you're ahead if you agree it doesn't match the question.

[Dan Byström]

This will take me quite a bit of time to wade through, but this seems to be exactly the sort of stuff I'm trying to pick up! Although I seriously doubt the "Pretty simple" part! :-)

[k00k]

Heh, I meant that the scripting part is the easy part. :)

[Dan Byström]

I can't see that they provide an API I can use myself? "FXhome" has software with supprt for a "hotfolder" but unfortunately that is not good enough. Luckily - I've been making huge progress on my own, but I'm not quite there... yet...

[Mike Gledhill]

After reading this thread (and having used Paint Shop Pro for many years), I realised it was dead easy to write my own basic "green screen remover" webpage, which then allows you to save the result to a .png file. This is much easier than using PSP !! mikesknowledgebase.com/pages/Other/ImageBackgroundRemover.ht‌​m

[Dan Byström]

Hi Gabriel, I wrote this question six years ago, but apparently someone has put a bounty on it - I hope he'll comment on your effort. I have had an acceptable solution for this problem running in production for many years now. You have provided a lores image with heavy compression artifacts, so it is not representable for the domain I'm working in, but some day I'll run your code on a test image of mine to see the difference. The thing that I’d like to solve nowdays, however, is how to distinguish between the background and cloth in the same color, but that’s another story. Nice work - cheers!

[Gabriel Ambrósio Archanjo]

Dan, do you have a frame of a person wearing a cloth of the same color? Send it to me: gabriel at garchanjo dot com. Let me take a look. Perhaps I can help.

[Dan Byström]

I'll see if I can dig up someting. PS. I accepted your answer temporary since it was the best so far - so you would get your 100 bounty. Merry Christmas.

[Mustafa sabir]

Can we use similar code for background removal from videos? What is the performance of this code? I think converting videos into individual photo frames, processing them and then re combining for background less video would be too resource intensive.

[Gabriel Ambrósio Archanjo]

Yes, you can! You can even remove background in more complex situations like this: youtube.com/watch?v=bwEGeI7E0n0

[John Nagle]

I tried that approach. The Reduce Green part is good for white hair, but not for much else. It bumps up the R and B channels a bit when the hue is in the greenish range. This is not helpful for greenish edges on dark areas. Reducing G to max(R,B) is a bit more useful; it takes out the green tinge,but may introduce grey. This step really needs to be edge aware, not single pixel.

[Gabriel Ambrósio Archanjo]

@JohnNagle Yes, you're right. We could use the morphological boundary algorithm to take the edges of the person shape and just apply these rules for pixels at a maximum distance of the edge. However, perhaps there are better ways to solve this. My solution is just a starting point, something relatively simple that can fit in a single page of source code. I'm pretty sure commercial solutions for this problem uses a much more sophisticated approach. Thanks for sharing your idea. It's a good next step improvement.

[c4da]

@GabrielAmbrósioArchanjo are there any resources for the if((r*b) !=0 && (g*g) / (r*b) >= 1.5) ? Not sure how or why did you came up with this? Is that just trial & error in order to treshold this particular image?