How to use wp_send_json_error?

wp_send_json_error( 'Error: Invalid data!' )

Will echoes a JSON string:
{"success":false,"data":"Error: Invalid data!"}

The nice thing about wp_send_json_error() is, the parameter could also be a WP_Error object.

As opposed to wp_send_json_success( 'Everything okay.' ) which echoes this JSON string:
{"success":true,"data":"Everything okay."}

Both rely internally on wp_send_json() to echo the JSON data properly and die() afterwards.

But if there is an error, then nothing gets returned.

Actually, I can’t believe this right now. But when I look into your JS code, you do not handle the JSON response properly. Actually if everything works fine, you return HTML, which replaces $('#submitYouTubeForm'). But with the JSON response, you do not have HTML but a JSON response.

Use console.log( response ) to see, if you really do not get the JSON string. You should think about either using JSON for all your responses or instead of using wp_send_json_error() just return a plain HTML error message and die() by yourself.