It seems to be safe, since most of the hosting companies around the globe that offer LAMP (Linux, Apache, MySQL, PHP) hosting services are providing it with WP-CLI already installed.
That said, generally speaking, WP-CLI is nothing but a bunch of WordPress functions implemented to run from the command line.
So, I understand that everything you can do with WP-CLI, you could also do without it, as long as you have SSH access to your server.
At the end of the day, I believe you have to take care of your SSH access policies more than WP-CLI.
And even if you decide not to install WP-CLI on your remote server, you can install it on your local machine and use it with ssh parameters to run commands on you remote server 🙂