Had Claude perform an audit of the tracking code. Here’s what needed improvement in my code:
- Hand tracking states flicker constantly but only
HandEffectsignored unknown tracking states—HandForcedid not, so every drop toTrackingState.Unknowntriggered the ball to get pushed for a moment. - Bodies got labeled as “lost” and immediately destroyed after going untracked for just one physics step. When the Kinect found the person again it gave them a new tracking ID, so they got a brand-new player and had to activate again.
- Nothing was being done to smooth inferred (guessed) or untracked joint positions, causing the jumps I was seeing on the body skeleton.
useTrackingStateColorswas broken because only the first and final keys of the joint prefab’s line gradient were getting replaced.AlignAndCalculateVectorshad a bug causing the right hand’s aim vector to use the left hand tip. Neither hand tip was ever placed by the Kinect, so the “aim” was really a small constant pull toward the camera. I swear this used to work.
Here’s how these things got fixed:
- Unknown, NotTracked and Lasso states are ignored and keep the current state. To change state, the camera now needs to register 2 consecutive frames in agreement, so single misreads are ignored.
- Hand state tracking now runs at the Kinect frame rate of 30 Hz rather than the physics step of 50 Hz.
- A lost player is frozen at its last pose for 1 s, and its ball keeps simulating. If a new body appears within 0.5 m of where the player was lost (measured at the base of the neck), it takes that player back over.
- Tracked joints are used directly, while inferred joints are smoothed over 0.1 s and untracked joints hold their last position.
useTrackingStateColorsgives each bone a plain two-color gradient, so colors cover the whole bone.- Each hand now aims along its own wrist-to-fingertip direction.
- Joint data used to be fetched 16 times per player per step (each fetch builds a new dictionary); it’s now fetched once.
It’s really a night-and-day difference.
Thing is, I also learned some very important limitations to the Kinect V2 during this audit. That I never learned about these earlier can only be explained by the adage ”Everything happens when it’s supposed to.”
Firstly, the camera’s color stream halves from 30 FPS to 15 when low exposure is detected. This explains why the camera performed so poorly when I demoed indoors at the Frost Planetarium’s dimly-lit lobby area. I need to add another camera viewing mode besides color stream that doesn’t get framerate restricted. I’m thinking that I could use the always-at-30-FPS infrared feed and inject some sort of color into it. At proper exposure, the color stream is too bright anyways and detracts from the visibility of the particles.
Secondly, according to multiple documentation sources, the Kinect V2 only supports hand state tracking on two bodies. The plan was already to migrate to a new tracking system once I start working on the next iteration of this project (V4) anyways, so no biggie, I guess.
Lastly, I had Claude fix Unity’s integration with Zed IDE by making some mods to the Zed for Unity package. I’m not sure why I switched to Zed, but since I did, I might as well make it work.
Tags: kinect hand-tracking body-tracking multiplayer performance physics installation