Since we can only get remote players' bone data at the moment, it's essentially impossible to create many different systems reliably (the most significant to me being any sort of reliable custom VRCObjectSync alternative optimized for any given world's specific needs) without some pretty significant compromises. You can either follow bones exactly without any offset, leading to awkward/incorrect visuals, or you can calculate local offsets before syncing data which leads to inconsistencies, or you can add arbitrary predefined offsets which also lead to inconsistencies, all while dealing with inconsistent latency offsets that can sometimes lead to wild discrepancies in position & rotation between the holder and remote users (syncing TrackingData on a manual behavior currently sends the data WAY too early, and there isn't really a way to verify it's still valid anyway since so many bone data points are just discarded to the point where creators can't reliably predict what will happen when a behaviour inevitably receives TrackingData too early). This is especially critical if the rotation of a synced object MUST be perfectly accurate at all times for any given use case; achieving behavior like this while also significantly reducing overall network usage and allowing creators to put manually-synced scripts on pickups (while also syncing their positions/rotations manually), is currently impossible without significant unnecessary sacrifice. This has other implications as well; if a creator needs to reliably put something above a remote user's head, rotated a certain way, they can't. Because "what if the remote user's avatar has their head rotated in an unconventional way?" "What if the rotation of a synced object is critical, but the remote player's avatar is culled, thus preventing Udon from getting accurate bone data?" "What if someone hides a remote player's avatar, thus likely changing all of their bone orientations while they're being tracked?" There are tons of reasons why only providing bone data just isn't sufficient at all for really anything that needs to be 100% reliable like TrackingData would be. Every player-tracking idea I can think of is always thwarted by "but what if a player has an unconventional avatar"? or "but what if a player's avatar is culled/hidden?" As a result, please allow GetTrackingData() to return remote users' actual TrackingData, not just bone data, in the same update/network timing as each player's bone data is sent, so that creators can create systems that reliably line up remote user TrackingData with their visible body movements; systems that will significantly reduce your servers' overall network usage fundamentally across the platform, while tangibly improving player experience. If this functionality needs to be opted-into so as to keep the default behavior performant by not calculating/syncing TrackingData all the time, that's fine; maybe have a synced boolean flag on VRCPlayerApi that reliably enables/disables the calculation/syncing per-user at manual sync timing with "OnTrackingDataSyncToggled()" callbacks. Or if you're already syncing this value alongside bone data, just expose it to us. Or if not, maybe you can calculate certain offsets for each possible avatar a user could present to a remote user at any given time (including while culled and while changing avatars, please make this available and accurate under those conditions as well), and save those offsets locally to be used whenever GetTrackingData() is queried. Even a compressed version of TrackingData that's localized to each player's playspace origin (or some other position that's consistently very close to the player, while also being the same for each player in the instance regardless of avatars or culling settings) that would cut the required networking data down to less than half using shorts/bytes (any Quaternion can be synced using 6 bytes, and the same can be said about any local-space short-distance Vector3) sent as needed, with a "remote user TrackingData may not be perfectly accurate to a tenth of a micrometer" disclaimer to creators, would be sufficient. Whatever the required/optimal solution may be, I'm sure the creator community AND the player community AND the VRChat server rack community, would all greatly appreciate this feature and the creations it would enable.
