KABE is the successor of Cinder Climb, my environment for interactive games on a climbing wall (or any wall). The first Cinder Climb tests were recorded on 11 March 2016 and used a Kinect 2 depth camera. KABE drops the Kinect: it works with an ordinary camera and finds people with a pose-detection neural network.
How it works#
- A camera watches the wall. Each frame is cropped to a region of interest and passed through YOLOv8 Pose, run with ONNX Runtime (on CUDA when available, otherwise on the CPU).
- The detected body keypoints are smoothed over time, weighted by the model’s confidence, and joined into 19 “bones”.
- The bones are drawn as thick polylines and merged into one body silhouette with Clipper2 (union, inflate, simplify).
- A four-point homography maps the silhouette from camera space to projector space, so what the projector draws lines up with the real climber.
- The activity itself is a Lua script with
setup,update,drawandexithooks. It can call openFrameworks drawing functions (through SWIG bindings) andcollide(x, y, radius), which says whether a point on the wall touches a person.
The program runs three threads: the GUI for configuration and preview, the projector renderer, and the inference worker.
Activity ideas#
Some of the planned activities are meant for people with disabilities, for example putting out points along a line or following the hands of a clock. Others are games for any climber: escaping rising water, catching eggs, or a “jumping frog” that hops from hold to hold.
Tech: C++, openFrameworks, YOLOv8 Pose, ONNX Runtime, OpenCV, Clipper2, Lua, vcpkg.