Analysis of the Kinect For Windows SDK 1.8 Background Removal API

An exciting new feature called the background removal API has been added to the Kinect For Windows SDK 1.8 which was released last week. Background removal or green screening is a feature many people have been using the Kinect for to varying degrees of success. Prior to the background removal API, getting a decent green screen effect out of the Kinect required a lot of heavy lifting and creative thinking because the depth data from the Kinect is too noisy to use for a smooth player mask.

A photo taken using the default kinect depth data with no blur or special techniques.

The official Kinect SDK 1.8 Background Removal API is a great step forward for developers as it allows obtaining a great green screen effect with minimal work. However there are a few important restrictions in place which make it unusable right now for a multi-user photo experience.

1. The initial Background Removal API requires successful skeleton tracking to work.


2. The initial Background Removal API only allows the effect to be preformed on one tracked player at a time. Update: 10/3/2013 - Joshua Blake writes in from Twitter, "By the way, you can do multiple people background removal in 1.8. You just need to create an instance per tracked skeleton.". Thanks Joshua.

Those criticism's aside, it would be awesome if Microsoft were able to decouple the Background Removal API from skeleton data or add a mode where you can specify depth away from the camera as a threshold instead of detected skeletons.

















A photo taken using the Kinect SDK 1.8 Background Removal API.

This summer for the Kinect Green Screen Photo kiosk I had at Maker Faire Detroit, I invested about a month of time figuring out how to get a good mask out of the Kinect in real time. The approach I used was to ignore skeleton data, and only use depth data, then run that raw data through EMGUCV(.Net OpenCV) and do blob detection, take the detected blobs, and run them through a point by point averaging algorithm based on work done in openFrameworks. I also used a simple shader based blur effect available in Windows Presentation Foundation as it proved way faster than any other implementations I tested as well as writing my own box blur or gaussian blur. While the results I came up with are not as good as the official SDK implementation they are pretty close and don't require detected skeletons. (The project is open source, so you can check out my implementation here: https://github.com/transcendingdigital/MFDetroit2013_Kinect_GreenScreen_PhotoKiosk) Not requiring skeleton detection is a HUGE factor when groups of users are posing for photos.

















A photo taken using custom techniques and EMGU CV.

How I think Microsoft's Background Removal API Works

The unfortunate part of the Microsoft Background Removal API is that it is closed source; so people like me who have been working on the same problem are interested to know exactly what is going on at a low level. I wish they would release a paper on the technique being used. Based on my own work there are a few things I am going to guess Microsoft is doing.

1. I am pretty sure they are using some sort of frame averaging of the depth data. During my own work I found this concept presented by Karl Sanford in his early work smoothing depth data. In my testing, averaging the depth data was too slow of a process in managed C# code and the results were not very good for creating smooth masks that fit the contour of subjects; so I threw out this technique. The tell in the 1.8 SDK that this is happening is when you wave your hands around or move fast, you can see some lag in the mask as it follows you; which could either be from frame averaging or intentional slow down of processing to increase end user performance.


2. I am confident that they are using external computer vision libraries that may have some licensing restrictions preventing them from being included in the official Kinect SDK DLL's. You can tell this because the background removal API is oddly contained in a 32 and 64 bit separate DLL from the main Kinect SDK dll. If you use EMGU CV, it requires individual 64 bit and 32bit dlls because it reaches into low level unmanaged code for much of the functionality it provides. I'm willing to be that the background removal API takes advantage of unmanaged code to speed up real time processing.


3. The requirement of including the skeleton data may only be used to know where to place the mask and may have no benefit on Microsofts background removal. In my own implementation the skeleton data is useless because it does not provide any detailed contextual data to help hug the contours of each humans body. It only provides stick figure like data.


4. I am guessing that the official 1.8 SDK background removal is limited to one player due to performance barriers. I don't know what the internal allowed CPU usage is for the Kinect SDK, but I would imagine Microsoft attempts to keep it low enough that developers do not have a problem integrating the SDK into end applications. Background removal requires frame by frame processing which is very processor intensive.


5. Microsoft spent time massaging the depth data provided by human hair and the head region. In the default depth data from the Kinect, human hair is always an issue causing the mask to really degrade some times on a persons head. This makes sense because the Kinect depth data is provided by an infrared grid that only has a resolution of 320x240.


6. There is some sort of blur going on, but it doesn't look like a conventional blur. Take a look at the edges of the photo from the background removal api. They have weird gaps in the pixels almost like some of the data is being removed for faster processing, or a better attempt at soft refinement of the edge of the mask.

I hope this article has provided some insight into background removal not seen elsewhere. I am looking forward to the next version of the Kinect hardware and appreciate Microsoft's continued commitment to the Kinect for Windows SDK.