Just set up my new Kinect camera. I was having some issues with it constantly restarting and found my way back to that trusty Reddit thread where I first learned that disabling the Kinect’s microphone can cause it to power cycle. The mic wasn’t disabled this time, though, but after perusing some of the thread’s comments, I found a second mic-related tip about disabling Windows’ default audio enhancement for the Kinect’s mic, and sure enough, that fixed it.
It’s finally time to add depth-based occlusion to the particles.
Prompt
The unity-cli bridge is running on port 6401 and the EnergyBall-V3 project is open to the main “Energy Ball V3” scene. This is the scene I display in production that draws the energy ball VFX particles on top of a live camera feed from the Kinect.
I want my particles to disappear when they’re behind a tracked person’s body. As far as I can tell there are two ways to do this. I might be slightly off in some of my methodology but I’m hoping you can reason through any of my logical mishaps.
The first way is using a grab pass shader to draw an “invisible” point cloud directly on top of the camera stream that’s perfectly aligned with the camera aspect. Point cloud pixels would derive their color from the corresponding the camera feed and the vfx particles would be drawn on a separate layer.
The second way is by passing a buffer of all point cloud particle positions into the vfx graph and comparing the vfx particle position to the point cloud particle position, and if the point cloud particle would be drawn in front of the vfx particle relative to the ortho camera, then we set that particle’s visibility to 0 in the update context in vfx graph.
Which way do you think is better? Consider from an implementation feasibility standpoint while also considering performance. Also, take a look at the logic I’m using to align the camera feed with the tracked bodies and particles currently (including the
zDepthproperty in the vfx graph), because the alignment isn’t perfect and I’m sure we should improve it as part of this new feature implementation.
The reason I recommended fixing the skeleton alignment is because I’d rigged together all sorts of manual positioning logic in order to align the tracked body skeletons with the camera view, getting things to a point where they almost looked right; I knew it wasn’t optimal, as you can see from the pic below.
Claude confidently recommended choosing option 1, but explained that a grab-pass wasn’t even warranted: “If you render a depth-only occluder (a ColorMask 0 shader that writes depth but no color) shaped like the person’s body on layer 6, any particle pixel behind it gets z-rejected, and the pixel that shows through is the base-pass video — which is literally the person. The occluder never needs to reproduce camera-feed colors, so the whole grab-pass/‘sample the feed per point’ machinery drops out”. I had it go ahead and implement, and it worked right away.
Not without some bugs, of course. Mostly related to culling. For example, the skeleton lines were getting culled by the body’s point cloud instead of being drawn on top. As I tested more, I added new debugging views like the ability to toggle on a visible point cloud and a visible metaball box boundary.
Since the sphere gets drawn to the midpoint between the hands, I had to add new logic to prevent it from disappearing into the torso when the arms are outstretched at a person’s sides. I knew I needed some sort of forward offset with a falloff. Claude created this beautifully and simply, measuring the difference between the sphere target position and shoulder joints’ Z-depths and pushing forward according to values from two new parameters, torsoMaxForwardOffset and torsoOffsetFalloffDistance (distance from torso where the effect dies out).