How to use endpoint, but remove/rewrite endpoint base?

I think the add_rewrite_rule is the correct route to go and I think what you have is correct also barring the Regex.

Try substituting what you have currently for this => ^my-page\/([0-9]+)\/?.

Full code below:

function setup_filter_rewrites(){
    add_rewrite_rule('^my-page\/([0-9]+)\/?', 'index.php?pagename=my-page&my_var=$matches[1]', 'top');
}
add_action( 'init', 'setup_filter_rewrites' );