Like you suggested, use WP_HTTP_BLOCK_EXTERNAL to stop all external URL requests. And then use WP_ACCESSIBLE_HOSTS to set allowed URLs.
From the WP Codex, found on this page.
wp-config.php
define( 'WP_HTTP_BLOCK_EXTERNAL', true );
define( 'WP_ACCESSIBLE_HOSTS', 'api.wordpress.org,*.github.com' );
Block external URL requests by defining WP_HTTP_BLOCK_EXTERNAL as true and this will only allow localhost and your blog to make requests. The constant WP_ACCESSIBLE_HOSTS will allow additional hosts to go through for requests. The format of the WP_ACCESSIBLE_HOSTS constant is a comma separated list of hostnames to allow, wildcard domains are supported, eg *.wordpress.org will allow for all subdomains of wordpress.org to be contacted.
Note: This could have unintended consequences like potentially breaking plugins, auto updates, etc.