If you just want to delete Yoast SEO manually so you need to login into your server and delete the following folder:
.../wp-content/plugins/wordpress-seo
So you will delete the directory wordpress-seo
.
If I were you I’d do it via SSH as Yoast SEO has approximately 600 files.