First thing is: a good plugin should always remove its trace when uninstalled completely.
Any changes, whether plugin install, enabling new feature etc. should not be done on live server. We developers always do tests on a local server. And a good one for anybody could be Local by Flywheel.
If you are desperate to test things on live server, always, I repeat always, have a solid, tested backup before proceeding. Because you don’t know [until thorough code-reading] how the plugin/theme is coded.
Now let’s answer your question:
is possible that when switching from plugin to plugin some files from the old one remains and creates conflicts (and security issues) with the new plugin?
If and only if your active plugin is a good one, it will remove its trace on uninstallation. And thus it’ll clear its traces while removed.
Then and only then you can assure yourself that, you can have a clean install of a new plugin without conflict.
But typically popular plugins conflict less than other. So you can have a try, after having a strong backup.
Switching between security plugins is a risk?
And switching security plugins is never a risk because both are doing their job to secure you, if they really did what they really are saying. If you are curious about plugins’ traces, have a good read on this thread:
Disabled plugins are they security holes – rumor or reality? — WordPress Development
And the bottom line is: Is your plugin coded good enough to trust it?
Update
And I’d love to recommend Ipstenu’s article about security through obscurity:
- Still Not Using Plugins for Security – Mika Ariela Epstein