No, you cannot.
You can add extra controls to a blocks toolbar or inspector because filters were added to the code that constructs those user interfaces. Think of it as being handed lego buildings before they’re placed as an opportunity to add extra bricks.
What you’ve asked however is filtering individual components, which is not possible. Unless a slot/fill was implemented, or a filter added, you cannot swap out the components like that at runtime.
But if you could, it would not be a good idea. The WP components library was built to provide generic atomic user interface building blocks with no logic behind them. That’s why you have a button component as well as a publish button component. The button component is just a dumb button that does nothing, it’s the wrapping component that adds the logic and integrates it into the user interface. Even a raw panel component doesn’t have open/close logic to toggle itself and requires more code to glue to together.
Your end goal might be possible, but the solution you dreamed up and asked about here is not it.