Removing the background in a picture

With a single line of code

With only one line of code, the rembg package finds the object of interest in a picture and removes the background. Or, at least, that is what it promises. But does it hold up to that promise? tags: U2-Net, image segmentation, image classification, deep learning, artifcial neural network

The tool

I recently had to upload a picture for an ID card. Now, I am not sure about you, but I don’t normally have such pictures lying around. Taking a selfie is easy enough of course, but finding a place with decent light and a good background, well, not in our house. There is always the option to tweak the picture afterwards, but that takes time, and can be tricky with those pesky shadows in the background.

Coincidentally (or is it?), that same day I came across a tweet about this little gem called rembg, a tool to remove the background in images. Sounds simple enough, but this is without you having to tell the tool what to extract! Maybe it is me, but I find that pretty amazing ¹.

Rembg uses the deep learning algorithm U2-Net ². This is a deep network architecture for so-called salient object detection (SOD). In other words, for the detection and localization of regions of images that attract immediate human visual attention ³. To be honest, how those algorithms exactly work is beyond me ⁴. However, the examples and user cases on the U2-Net Github site are quite convincing as to what it can deliver.

And the result above does not disappoint. The background in the picture is fairly uniform, but because of the shadows on the wall, removing the background manually would still have been a pain.

Installing and using

Installing the software is pretty simple. Just download it and install the tool with pip: pip install rembg. After installing the software, you use the fool from the command line:

rembg p path/to/input.png path/to/output.png

Or, alternatively, the code below might give you better results. It did for all pictures I used when trying out this tool.

rembg i -a -ae 15 input.png output.png

You can also use it as a server or as a library in Python. See the website for further instructions.

A few more examples

Here are some other examples of pictures from which I tried to remove the background using Rembg. And, I must say, overall, I am impressed ⁵. In all six pictures, the background is fairly noisy, but the software still manages to isolate the main objects in the picture fairly well.

The two results below are less convincing. It did not manage to recognize the background within the ear of the cup. But where it really fails epically is with the detection of the chameleon. Which surprises me given how clearly the chameleon stands out from the background. For some reason, rather than distinguishing the chameleon as one object, Rembg singles out the two body parts with a deviating color.

But, overall, the tool does a pretty good job. And of course, you can train and fine-tune your own model if this isn’t good enough (see the website for some clues how to do this).

Footnotes

1. I know that applications based on AI and deep learning models are becoming increasingly commonplace in our lives. I also know that my children tend to take a lot of this for granted. However, I think we should never stop marveling at what these techniques are capable of, or being critical of their limitations and risks.

2. Qin, X., Zhang, Z., Huang, C., Dehghan, M., Zaiane, O. R., & Jagersand, M. (2020). U2-Net: Going deeper with nested U-structure for salient object detection. Pattern Recognition, 106, 107404. https://doi.org/10.1016/j.patcog.2020.1074042

3. Gupta, A. K., Seal, A., Prasad, M., & Khanna, P. (2020). Salient Object Detection Techniques in Computer Vision—A Survey. Entropy, 22(10), Article 10. https://doi.org/10.3390/e221011743

4. Image recognition and object detection in remote sensing and biodiversity monitoring are two focus areas of our Innovative Biomonitoring research group. As such, I do try to keep up with the different tools and methods. But with all these fast-moving developments, it definitely is a challenge.

5. Note that I used the original images as input for the rembg. The pictures as shown were resized afterwards. In all cases, I used as command rembg i -a -ae 15 input.png output.png. See this site for what the -a -ae 15 parameters do.