KABE: an interactive climbing wall

KABE is the successor of Cinder Climb, my environment for interactive games on a climbing wall. The first Cinder Climb tests were recorded on 11 March 2016 with a Kinect 2 depth camera, and KABE drops the Kinect to work with an ordinary camera and a pose-detection neural network that finds people in the picture.

How it works

#

1. A camera watches the wall. Each frame is cropped to a region of interest and passed through YOLOv8 Pose, run with ONNX Runtime on CUDA when available, otherwise on the CPU.
2. The detected body keypoints are smoothed over time, weighted by the model’s confidence, and joined into 19 “bones”.
3. The bones are drawn as thick lines and merged into one body silhouette with Clipper2.
4. A four-point homography maps the silhouette from camera space to projector space, so what the projector draws lines up with the real climber.
5. The activity is a Lua script. It can call the openFrameworks drawing functions and collide(x, y, radius), which says whether a point on the wall touches a person.

The program runs three threads: the GUI for configuration and preview, the projector renderer and the inference worker.

What it can do

#

Several climbers can be on the wall at once. Every detected person gets their own silhouette, and a script can ask how many people the camera sees, and whether a circle at some point of the wall overlaps any of the bodies (for one point, or for a whole list of points in a single call).

To keep tracking stable, keypoints are blended with the previous frame and low-confidence points are ignored. Silhouettes disappear only after a full second without detection, which means missed frames won’t make them flicker. You don’t have to buy a depth sensor or use special markers for any of this, since an ordinary USB webcam is enough.

While working on an activity you can edit the Lua script, press the space bar or r, and the activity restarts without restarting the program.

Setup and calibration

#

All you need is a PC, a camera that sees the wall and a projector. The program opens two windows, one for the projector and a control window that shows the camera preview, the silhouettes and two FPS counters (inference and drawing), with a slider to select the activity.

Calibration is done with the mouse in the preview. First you drag four markers onto the corners of the area the projector covers, which defines the homography between camera and projector, and then two region-of-interest points to crop the part of the image that goes to the model (a smaller region means less work for it). Both are saved to cc-settings.xml after every change, so the next start needs no calibration. To check that the mapping lines up with the wall, there’s a red dot in the projector window that follows the mouse through the homography.

Writing an activity

#

Activities are Lua files in the scripts folder. Files named game_*.lua are picked up automatically, start.lua is always the first one, and you switch between them with the arrow keys.

A script defines setup, update, draw and exit, and it can also define key and mouse callbacks. A click in the camera preview reaches the script as a click at the matching projector position, which is handy when you want to test things without climbers on the wall. The script gets:

• machine:collide(x, y, radius), true when the point touches a person;
• collideAll(points, radius), which takes a table of points and returns a table of booleans, cheaper than many single calls;
• the person count and the screen size;
• the whole openFrameworks API, so images, shapes, fonts and sound are all available.

In the project notes I also listed a small set of Lua base classes (objects with position and fade, touchable shapes with a state machine, a collision manager, a Box2D wrapper and sprite animation) and two starter games, a box game and a points game.

Performance

#

According to the architecture notes, inference should run at about 30 frames per second on a CUDA GPU and about 10 on a CPU, with drawing at 60 frames per second and vertical sync. Those are expected figures, not measurements. The code was tuned for it: the ONNX input buffer is reused, the camera frame is double buffered, a texture is no longer created per frame, and the Lua garbage collector runs one small step every 30 frames. The notes put the overall gain at 45 to 70 percent, and say plainly that it’s an estimate.

Hardware and limits

#

• Windows 10 or 11 (64 bit), built with Visual Studio 2022 and C++20.
• An NVIDIA GPU with CUDA is recommended. A CPU run needs a code change and a rebuild.
• One camera and one projector. The projector size is fixed at 1024 by 768.
• The model path is fixed in the code.
• The source is in a private repository.

Planned activities

#

Some of the planned activities are meant for people with disabilities: putting out points along a vertical line, one-sided and alternating, putting out segments of a picture around the whole wall, sorting elements by category, and following the hands of a clock. The others are games for any climber: escaping water that rises from below or fire from above, erasing a picture with your body, catching eggs (the higher, the more points), smearing paint with the whole body for children who take their first steps on holds, matching a shape shown on the wall and holding the pose, and a “jumping frog” that hops from hold to hold.