The way I see it, players you can't see or hear shouldn't have any noticeable impact on performance, but currently that doesn't seem to be the case. Framerates consistently plummet in instances with a large amount of users, regardless of whether or not you're in the same area, room or building as everyone else.
My guess as to why this happens (aside from complicated world gimmicks) is that things like IK, voice and various whitelisted avatar components are still taking effect for every user. I don't know how feasible it'd be to set up some kind of automatic or manual culling system, but something as simple as locally blocking players who aren't in your line of sight or the world's defined voice falloff range could be a godsend for creators who like to make big, expansive worlds with multiple hangout areas.
To clarify, "culling volumes" are already a thing, but they only really help cut down on draw calls.