Where do I put my add_action(… and add_filter(… and do I need to remove them?

filters and actions are added with every request. when you disable a plugin, they immediately disappear, because they only exist in your plugin code. this is how you’re able to selectively add them on a per-request basis. you only need to remove filters or actions if you don’t want them to run in the current request.

is_admin() is a check if the admin pages are being displayed, and should wrap things only for admin pages.