OnDeserialization gets called for all non-owners every time a player joins.
I made this world to demonstrate this. Have two people in the world and then have the master click the cube and the non-owner player will see the count go up. Then have a third player join the world and despite the master/owner not clicking the cube to change the synced variable OnDeserialization gets called again for the non-owners in the world causing the count to go up.