How it works

#

A Kinect 2 depth camera looks at the wall. The program keeps a model of the empty wall and subtracts it from every depth frame, filters out the noise, and whatever blobs are left become outlines of climbers.

To match camera to projector there’s a calibration window in which you drag four markers and two region-of-interest points, and the program saves them for later runs. Activities are Lua scripts that draw with the projector and keep asking the program whether some point on the wall touches a climber.

Activities

#

You switch between activities with arrow keys, and the space bar reloads the current script without restarting the program. Scripts have access to the openFrameworks API, Box2D physics and the Kinect data: how many blobs there are, what their outlines look like, and whether a point lies inside one of them.

• Coins. Balls fall down the wall under Box2D gravity and you score by touching them with your body. Balls that reach the floor are misses. A round lasts 60 seconds, and gravity gets stronger with every level.
• Position. You see a picture and have to fill its shape with your body, up to a 60 percent match.
• Totoro. Your silhouette works as a mask that uncovers a picture chosen at random from a set.
• Trail. Silhouettes are painted on the wall and fade slowly, leaving trails behind climbers.
• Box. The wall turns into a grid of 32 by 16 tiles that light up when you touch them, and their colour changes every so often.

Limits

#

It runs only on Windows with a Kinect 2. Because everything depends on that model of the empty wall, you have to reset the background whenever something in the scene changes. Game texts in the scripts are in Polish.

What came next

#

The project continues as KABE, which replaces the Kinect with an ordinary camera and a pose-detection neural network.