Allow mirrors to use ARGB Half RenderTextures for correct post processing
complete
Merlin
Please add an option to use ARGB Half render textures for the mirror’s target render textures instead of the ARGB 32 textures they currently use.
This allows mirrors to get proper tone mapping and bloom applied to what they render which allows easy and complete parity between what you see in a mirror and what you see in a world when using the post processing stack.
I have attached an example of a particularly bad case for mirrors. The first picture shows the scene and what it should look like, along with the camera positions used for the demo. The second picture shows a comparison of what happens with each texture format. The third picture is an in-game capture to demonstrate the same color saturation happening on mirrors. The first two pictures are done using cameras pointing to render textures and quads sampling from those render textures for their emissive channel since mirrors aren’t available in editor.
The left quad is using a render texture of ARGB32 in the same way that mirrors do. This causes the cameras to do the HDR to LDR resolve without any decent tonemapper applied or bloom filter which causes the colors to saturate before they can even make their way to be rendered on the mirror quads. The right quad is using ARGB Half as the format which prevents the colors from getting clamped before the mirror quad gets rendered. This allows the color values >1.0 to make their way to the reference camera’s post processing stack correctly and have proper tonemapping and bloom applied.
This is pretty much a 2 line fix. You just need to add a public bool useHDRTextures; or something and wherever the render textures are created you just and pass something like RenderTexture mirrorRenderTex = RenderTexture(renderTexWidth, renderTexHeight, 24, (useHDRTextures && mirrorCam.allowHDR) ? RenderTextureFormat.ARGBHalf : RenderTextureFormat.ARGB32);
Performance characteristics:
Tests done on a GTX 1080 ti @ 4098x2048 render texture resolution
With MSAA disabled on the render targets you actually get better performance by omitting a blit from an internal ARGB Half texture to the bound ARGB32 texture (though you can also get this benefit by disabling HDR on the mirror camera for ARGB32). With MSAA you do get worse performance though.
No MSAA ARGBHalf: 1.477ms
No MSAA ARGB32: 1.693ms
2x MSAA ARGBHalf: 2.321ms
2x MSAA ARGB32: 2.22ms
8x MSAA ARGBHalf: 5.672ms
8x MSAA ARGB32: 3.269ms
Log In
Aev
complete
Added in VRChat 2019.1.2!
Mr Q
in progress
Merlin
I spent some more time today investigating and verifying some odd behavior I saw when I originally made the canny. I want to do a few more profiles, but at this point I can say with decent certainty that switching the target texture to ARGB Half will be a win for performance and vram on all maps.
The grab pass crash was brought up by Nepsy when I was making the canny so I tested it with ARGB Half textures and an avatar with some grab passes on it. ARGB Half does not cause the crash. It looks like it only happens on 128bpp formats like ARGB Float and ARGB Int as far as I can tell at the moment. As for the memory thing I wasn't too clear about it at the end of the original post, but making the mirrors use an ARGB Half render texture is a win for vram as well as the gain for omitting the blit. When HDR is specified on the mirror camera and the render texture is on ARGB32, it’ll create both an ARGB Half and ARGB32 render target. With using ARGB Half instead you’d actually be saving 33% vram since it doesn’t need to create an intermediate texture.
When you use an ARGB32 render texture on a camera marked as HDR, Unity will do some work under the hood for you. Unity creates a temporary ARGB Half texture and then renders the scene to that texture. Once the scene is rendered to the ARGB Half render target Unity will run an MSAA resolve on that texture, then blit the resolved ARGB Half texture to the ARGB32 texture that the camera specifies as an output. Then finally if the render texture has an MSAA >1 it’ll run a second, redundant MSAA resolve that doesn’t affect the image output at all. This second resolve is what the mirror MSAA setting would affect.
The interesting thing is that the hidden ARGB Half texture Unity creates uses an MSAA level specified by the client’s rendering settings, not the render texture. This means that if your game is running at 8xMSAA, mirrors will also render at 8xMSAA regardless of the mirror setting. This is probably the case for all worlds. Going to my world with the reference camera set to HDR enabled vs disabled makes no difference and the mirror will render at 8xMSAA when my client is running at 8xMSAA. Lag free box also renders its mirror at 8xMSAA to an ARGB Half render texture internally even though the map has HDR turned off.
error.mdl
There's a very good reason for them using ARGB32 on mirrors and not any other color format. In the version of unity VRChat currently uses there is a bug with grabpasses and other color-format cameras that causes an instant crash. Grabpasses are a shader feature that allows a shader to access an image of everything drawn on a players screen up to a specified point. They are very widely used for effects like distortion. The moment a camera that writes to a non-ARGB32 rendertexture sees a material that calls for a grabpass, the game crashes for some reason.
Additionally, you also need to consider how much VRAM mirrors suck up. As someone who used to have a 970, I can safely say that mirrors can completely destroy that graphics card simply due to vram usage alone. ARGB half uses 50% more memory than ARGB32. While its no issue for high end cards with loads of VRAM like your 1080TI, the VRChat team needs to consider people that are on min-spec systems.
T
TCL
error.mdl: According to Unity a 2048x2048 (max mirror resolution) render texture is 32 MB for RGBA32, and 48 MB for a ARGB Half. This isn't going to make a difference, and based on what Merlin has said below, Unity's already using an ARGB Half internally anyways so getting rid of the RGBA32 texture should be a net savings.
owlboy
I always kinda liked how mirrors looked slightly different than the world (they were not 100% super duper perfect). But that huge color shift in your example is obviously not a thing were I think it's a nice "slightly off, but not too much" situation.
Maybe exposing this as an option the user could pick in the SDK would be nice?