Isn’t this supposed to be a job for the window manager?
For example, my virtualbox install has some mouse pointer bugs on wayland, and I can’t run waydroid on X. These things are weird to me. Shouldn’t window managers abstract all that for the software we run?
They are themselves abstraction layers for the apps that are made for them. Software has many levels of abstraction from what you see on the screen all the way down to hardware.