Just use a tunneling service like tailscale. Easy as fug to set up, and only people who know your credentials can poke about in your server.

If you remember to disconnect machines other than the server from the VPN when not using them and don’t share out the server too much, you don’t even have to spend money.