How to allow core Gutenberg blocks selection only when you are inside a custom inner block

You will need to filter the template for the post type:

add_action( 'init', 'setup_template' );

function setup_template() {
    $post_object = get_post_type_object( 'post' );
    $post_object->template = [ [ 'your/custom/block' ] ];
    $post_object->template_lock = 'all';
}

This will pre-populate the post with a single instance of your custom block and lock it from being changed.

By locking it, they cannot insert any top-level blocks but can still insert items into your custom block. There is more info in the docs about locking, you may need insert instead of all depending on your intent.

Hope it helps.