Added in another missing feature yesterday that I had meant to include—the ability to use my mouse as a kill switch to reset the scene. I had that code in the original project so it was as simple as copy and pasting it into my new one. Done.
I don’t think I explicitly mentioned this before, but the Azure Kinect Demos package dev, Rumen, responded to my query about hand tracking on the Azure:
Hi Joss, if the hand state tracking is critical part of your project, I would recommend you to stay with Kinect-v2. Let me explain. Azure Kinect Body Tracking SDK doesn’t track the visibility status of the body joints very well. That’s why you get a “tracked” state, even if your hand is behind the back. Moreover, there is no hand-state tracking in the SDK, although this was requested many, many times. I tried to replace it with a simple algorithm that analyzes the tracked fingers to estimate the hand state, but it’s not very reliable, due to unreliable finger tracking. A better solution would be to utilize a separate hand and finger tracking package for this. I’m going to implement this at some point in my CV-Examples package, but I’m still far from there.
Why the Azure Kinect Body Tracking SDK wouldn’t include hand-state tracking is beyond me.
For the purpose of demoing my project, I switched back to the V2, and although its skeleton tracking works decently enough, it leaves a lot to be desired. Quick user motions and underexposed/overexposed lighting conditions really mess with its skeleton tracking. The Azure had more accurate skeleton tracking, but no hand state recognition. In my estimation, a better setup would utilize the Azure’s skeleton tracking along with another realtime hand-state tracking library.
I’ve seen hand tracking libraries for Touch Designer on Instagram but I’ve never used any for Unity. A quick search points me to this repo and this one. That second repo seems like it could be of particular use in my project, since it “primarily focuses on the recognition of closed fist gestures for both hands.” Dope.
Tags: gamedev unity kinect hand-tracking body-tracking programming mediapipe opencv