Remote PlayerAPI GetTrackingData gives human body bones, not tracking data.
Phasedragon
This is problematic in a few situations, and this other issue makes it even worse: https://vrchat.canny.io/vrchat-udon-closed-alpha-bugs/p/playerapi-humanbodybones-do-not-have-a-consistent-rotation-every-avatar-is-diffe
1) It is, as far as I can tell, completely impossible to get the correct rotation of a remote user's head without syncing an object's position/rotation.
2) generic avatars will give you positions at 0,0,0
3) desktop users tracking data locally will give you hand positions out in front of the camera, in the position that the object would be if they were holding a pickup. This is a useful emulation and doesn't function for remote clients.
Log In
mahu
Phasedragon Since the tracking data and human body bones have always been inconsistent between remote and local clients and for different avatars, and we now have fallback avatar IK syncing to remote clients with some kind of normalized bone rotations for the human body bones, hopefully we're in a good position to bring some consistency to this API. If the rotations used for displaying remote fallback avatars and stuff can be calculated both locally and remotely and then provided as the rotations in udon that wouldn't be a breaking change to the API and then we would have useful body bone rotations and it would be simple to fix tracking data orientations for the remote clients hands as well.
Phasedragon
mahu: yes, this is something I've looked into quite a bit. It's not a trivial problem, because unity's humanoid bone system is a bit of a black box. fallbacks work by just applying the same "muscle" data to a different avatar, which results in a similar pose. But there isn't really a good way of directly taking that muscle data and turning it into a complete, standardized position/rotation unless you were to apply that muscle data to a standardized rig.
It is possible to use reflection to reach into the unity black box and pull out information about the current rig and the rotations needed to provided a calibrated position, but reflection is a very messy, expensive method which could break with unity updates. It's appropriate for editor scripts but better to avoid in a built game.
Another method that I've personally explored is to perform a calibration every time an avatar is loaded while they're in a T-pose, and store a relative rotation that you'd need to get from the actual bone rotation to a standardized rotation. Then when udon asks for a standardized rotation, you combine the two together. The problem with this is that not all T-poses are equal. Some avatars have overridden their T-pose to be an A-pose or even I-pose, usually for the purposes of changing how you calibrate FBT. But for the purpose of standardized calibration, that screws things up quite a bit and it would not be trivial to jam in a whole new playable layer just for a new T-pose. Additionally, some avatars have really weird bones that makes it hard to build a good standard without reaching into the unity black box. I have spent some time mostly handling that, but the more you look the more exceptions you find.
mahu
Phasedragon: Ahhh drats, that's much more cursed than I thought it was.
mahu
Phasedragon: You know, now that I think about it, I wonder if you could take the muscle data that is used for IK syncing and apply it to a standardized rig with the proportions matched to the avatar. Might be able to combine that with what you were trying to do: just load up the standard rig once during T pose, apply the same muscle positions to it that the T pose has, and then carry the offsets with the avatar. Although I don't know how the muscle positions and T pose and all that all relate so that is probably something you've looked into as well.
Phasedragon
mahu: applying muscle data to a standardized rig is a lot to ask if you do it every frame for every player because you're just duplicating the amount of work, which is already quite expensive.
But only doing that once on calibration and saving the difference is a very good idea. It would completely avoid the issue of people having A or I poses, while also providing a much more "normal" standard rather than something homebrewed. I'll have to experiment with that some time to see if it would work.