When playing for a semi long time, vrchat uses up to 6,5gb of video memory on my system (the max it can get).
Changing to an empty world does not free up any of that.
Would be nice if someone could confirm this on their system as well. In theory you would just need to go to a fairly full public instance, check vram usage, go to an empty instance, check vram usage again.