WordPress capabilities and restricted categories access

I think best hook should be 'template_redirect', when this hook is fired main query is already set, so you can look at queried object and if the user has no required capability you can redirect request somewhere:

add_action( 'template_redirect', function() {
  if (
    (
      is_category( 'special-category' )
      || is_singular() && has_category( 'special-category', get_queried_object() )
    ) &&
    ! is_user_logged_in() || current_user_can( 'special-cap' )
  ) {
    wp_safe_redirect( home_url() );
    exit();
  }
} );