Remove rewrite endpoint on deactivation?

All you need is flush the rewrite rules on deactivation. For example, in the main plugin file:

register_deactivation_hook( __FILE__, function() {
    flush_rewrite_rules();
});