The pattern you posted…
www.mydomain.com/image.jpg('custom-thumb');
… is a bit odd. That would be pretty tricky. You’d need a PHP handler to load the image and you’d need to tell the server (Apache, Nginx, IIS, Whatever) to parse that file ending as PHP. Something like this would be simpler:
www.mydomain.com/image.php?size=custom-thumb
You would still need to create a PHP handler script to read the URL, parse the GET string, and display the image.
You could probably get something like this…
www.mydomain.com/image/custom-thumb-X
… working with an endpoint.
However, the easiest thing to do is use wp_get_attachment_image_src to create the URL. I don’t know if that is an option for you but there is an example in the Codex:
<?php
$attachment_id = 8; // attachment ID
$image_attributes = wp_get_attachment_image_src( $attachment_id ); // returns an array
?>
<img src="https://wordpress.stackexchange.com/questions/116867/<?php echo $image_attributes[0]; ?>" width="<?php echo $image_attributes[1]; ?>" height="<?php echo $image_attributes[2]; ?>">