There’s a few ways to do this, but I would argue that the preferred way is, in general, fetching the post_title attribute from the post object. This does not depend on removing all filters for a certain function and adding them back later — the latter requires you to directly access the global $wp_filter.
get_post retrieves the post object for a post ID, and the post object, on construction, populates all post fields with the fields from the database, without applying any filtering.
Thus, your code would simply be
$title = get_post( $post_id )->post_title;
If the post with ID $post_id is not guaranteed to exist, be sure to check whether the returned value from get_post is a post object.
NB another approach is to use get_post_field( 'post_title', $post_id ), which by default only has the filter 'post_title' applied to it (and not the the_title filter). However, as @PatJ kindly pointed out, using the optional $context parameter, you can get the raw value using the context "raw":
get_post_field( 'post_title', $post_id, 'raw' );