remove_image_size doesn’t seem to be working

I don’t think it works that way. I think remove_image_size only affects image sizes created by plugins or themes, not the core ones in WordPress (could be wrong). If you want to use it to remove “custom” sizes you added, you will still need to hook it into an action to actually run it:

function remove_image_sizes() {
if ( function_exists( 'add_image_size' ) ) { 
    remove_image_size( 'custom-image-size1');
    remove_image_size( 'custom-image-size2');
 }
}
add_action('after_setup_theme', 'remove_image_sizes');

If you want want to remove core sizes, you’ll need to do something different.

function paulund_remove_default_image_sizes( $sizes) {
    unset( $sizes['thumbnail']);
    unset( $sizes['medium']);
    unset( $sizes['large']);

    return $sizes;
}
add_filter('intermediate_image_sizes_advanced', 'paulund_remove_default_image_sizes');

You can read about options in more detail here: https://andorwp.com/add-remove-change-and-regenerate-image-sizes-in-wordpress/