Yes
The overhead of a plugin sitting in your plugin folder is negligible, and would have an impact on the plugins page. The main area of optimisation here is the % of free space on your filesystem.
Active plugins on the other hand can run queries, make remote requests, do expensive parsing, filesystem operations, and all manner of expensive, longwinded, slow operations.
Not all plugins are expensive to run however. Hello Dolly is a simple plugin that does almost nothing, whereas a plugin that fetches RSS feeds and creates posts can be an expensive plugin when you give it a lot of feeds.
If a plugin is expensive to run, deactivating it fixes that. A deactivated plugin is just a file sitting on the filesystem, not being executed.
Is it worth getting rid of plugins you aren’t using? So long as they’re kept up to date, there’s no reason to remove them, but I would recommend keeping the plugins installed to those you actively use as not all plugins are maintained