This is the most likely answer. Leading vocals (and instruments) are normally centered, i.e. the same level on both channels, while background instruments are louder on one channel.
With the ground wire gone, you don’t have a stereo signal left-ground / right-ground anymore, but a mono signal left-right, i.e. you only hear the difference between the two channels. That cancels out any centered vocals and instruments.
The volume will also likely be much lower, as the signal has to travel through twice the resistance (two speakers) at probably roughly the same voltage.

This actually used to be a nice trick to get a pseudo-instrumental version of any song - just subtract the channels from each other. You’ll get a mono version of the song with only the background music. Not sure how it’s done today, there’s probably a better solution now.