Load WooCommerce templates from my plugin folder first

I think you can use template redirect action.

I don’t test it but it should work:

function my_template_redirect(){
   //pages you want to make true, ex. is_shop()
   if(is_shop()) {
       $plugin_dir = WP_PLUGIN_DIR."https://wordpress.stackexchange.com/".str_replace( basename( __FILE__), "", plugin_basename(__FILE__) );
    load_template($plugin_dir . 'your-template.php');
   }
}

add_action('template_redirect','my_template_redirect');