I deleted a material using a large texture and complex shader, and noticed that the size of my map didn't decrease at all. After some debugging I found out that the VRC_SceneDesciptor on the VRCWorld object still had a reference to the material in it's 'Dynamic Materials' despite no object in the scene using it. A few of the slots were also set to None.

Recreating the VRCWorld object removed the reference and the map was smaller now.