TL,DR;

You are hooking early.

In the plugin, the css file loads site-wide by default, I want to load
it only on the single post.

It’s a very good thought many developers don’t take this into account while developing themes/plugins.

is_single() isn’t working in plugins

Where are you using is_single template tag in WP run cycle ?

Assuming you are hooking to init or wp

Template tags are not available yet until template_redirect action. So those tags don’t work for init or wp.

Best place to work with scripts or styles is on wp_enqueue_scripts which runs after template_redirect .

Any idea how can we achieve that ?

There are many ways to achieve depending on the plugin we are using.

Coming to TablePress plugin.

controllers/controller-frontend.php is the place where plugin is en-queue-ing stylesheet.

if ( apply_filters( 'tablepress_use_default_css', true ) || TablePress::$model_options->get( 'use_custom_css' ) ) {
    add_action( 'wp_enqueue_scripts', array( $this, 'enqueue_css' ) );
}

You can completely turn off en-queuing the stylesheet using the tablepress_use_default_css and custom css option to false, and then load them wherever you need in your custom plugin/theme.

If you dig through the file there are many conditional en-queuing is done for that plugin.

You can also dequeue style first and then enqueue wherever needed.

Sample code

add_action('wp_enqueue_scripts','wp_228402_load_styles_conditionally',11);
function wp_228402_load_styles_conditionally() {
  if(is_single()){
    return; // return if not on the single post/page
  }
  wp_dequeue_style('tablepress-default'); 
}

Check the priority for the above that’s the key it should be greater than default value 10.