js alert in add_filter function for image_send_to_editor

You can try as like this, it may help you.

function filter_image_send_to_editor( $html, $id, $caption, $title, $align, $url, $size, $alt ) { 
    if('your error statement') {
      $html="<script>alert("Some Error Message")</script>";
    } else {
        //your continue code
    }
    return $html; 
}; 

// add the filter 
add_filter( 'image_send_to_editor', 'filter_image_send_to_editor', 10, 8 ); 

More Info Regarding image_send_to_editor Filter : http://hookr.io/filters/image_send_to_editor/