This is a big problem when putting colliders on your hands to do interactive triggers. Without putting the colliders on the MirrorReflection layer they will prevent you from interact (clicking) on objects, pickup objects and everything else that requires the "Hand Raycasts".
The "MirrorReflection trick" is not well known, unintuitve and restricts you from customising the collision matrix in this regard.
No idea what the best solution would be, but this needs to be adressed at some point imo.