How to show the category filter that’s shown on the ‘All post’ pages on a custom post type page in the admin area?

If your post type supports the category taxonomy, then that drop-down menu would be displayed by default:

register_post_type( 'my_cpt', [
    'public'     => true,
    'taxonomies' => [ 'category' ], // here, the post type supports the `category` taxonomy
    //...
] );

Otherwise, or for custom taxonomies, you can use the restrict_manage_posts hook and wp_dropdown_categories() to add that drop-down menu. Working example:

// $which (the position of the filters form) is either 'top' or 'bottom'
add_action( 'restrict_manage_posts', function ( $post_type, $which ) {
    if ( 'top' === $which && 'my_cpt' === $post_type ) {
        $taxonomy = 'my_tax';
        $tax = get_taxonomy( $taxonomy );            // get the taxonomy object/data
        $cat = filter_input( INPUT_GET, $taxonomy ); // get the selected category slug

        echo '<label class="screen-reader-text" for="my_tax">Filter by ' .
            esc_html( $tax->labels->singular_name ) . '</label>';

        wp_dropdown_categories( [
            'show_option_all' => $tax->labels->all_items,
            'hide_empty'      => 0, // include categories that have no posts
            'hierarchical'    => $tax->hierarchical,
            'show_count'      => 0, // don't show the category's posts count
            'orderby'         => 'name',
            'selected'        => $cat,
            'taxonomy'        => $taxonomy,
            'name'            => $taxonomy,
            'value_field'     => 'slug',
        ] );
    }
}, 10, 2 );

And in that example, you’d only need to change the my_cpt (post type slug) and my_tax (taxonomy slug). But feel free to customize the code to your liking..