I got curious, so I tested this barebone demo:
add_action( 'rest_api_init', function () {
register_rest_route( 'wpse/v1', '/post_by_permalink/(?P<path>[\S]+)', [
'methods' => WP_REST_Server::READABLE,
'callback' => 'wpse_callback',
'show_in_rest' => true
] );
});
function wpse_callback( $request ) {
$data = [ 'path' => base64_decode( $request['path'] ) ];
return $data;
}
Testing:
gives
{
path: "https://wordpress.stackexchange.com/q/301067/"
}
Tested also
that gave:
{
path: "https://wordpress.stackexchange.com/questions/301067/register-rest-route-regex-option-for-base64-or-alternate/301076"
}
Tested further:
https://example.com/wp-json/wpse/v1/post_by_permalink/dGVzdC90ZXN0LzM1NzM0Ly0=
with output:
{
path: "test/test/35734/-"
}