One of my plugins broke when I tried to update it, how do I safely uninstall it?

If you just want to delete Yoast SEO manually so you need to login into your server and delete the following folder:

.../wp-content/plugins/wordpress-seo

So you will delete the directory wordpress-seo.

If I were you I’d do it via SSH as Yoast SEO has approximately 600 files.