Load custom css after bootstrap

The above looks correct, however, you need to specify your script to enqueue after bootstrap by adding the bootstrap-min handle to the third, dependencies parameter.

function register_css() {
    wp_register_style( 'bootstrap-min', get_template_directory_uri() . '/css/bootstrap.min.css' );
    wp_register_style( 'custom-css', get_template_directory_uri() . '/css/custom.min.css', 'bootstrap-min' );
    wp_enqueue_style( 'bootstrap-min' );
    wp_enqueue_style( 'custom-css' );
}
add_action( 'wp_enqueue_scripts', 'register_css' );

See wp_register_style for more.

Alternatively you can, if you know you need to enqueue the files immediately from within your callback function, use wp_enqueue_style just like wp_register_style, see: wp_enqueue_style