Can some vulnerabilities in plugins be exploited even when the plugin is inactive?

Yes, of course. If the PHP code in a file works when that file called separately, and if there is a vulnerability, it can be exploited.

There are two basic rules. They apply not only to WordPress, but to every web site:

  1. Use only code that you understand completely.
  2. Do not put unused code onto your server.