Animators are disabled when an avatar is outside of your view, but IsAnimatorEnabled will stay true when it shouldn't. Distance-based culling works as intended, but frustum culling does not.
Video of me reproducing it (it was too big to upload here directly): https://www.youtube.com/watch?v=WwbYiJ5WUuA