Is there any action filter/hook for validating a custom field before publishing the post?

At the beginning of wp_insert_post, the function that saves/updates a post, there is a filter called wp_insert_post_empty_content. By default this filter checks whether the title, editor, and excerpt fields are all empty, in which case the save process will be halted.

However, since all the fields to be saved are passed to this filter, you can expand this filter to include any other test to determine whether the post should be considered empty. It would be something like this:

add_filter ('wp_insert_post_empty_content','wpse312975_check_unique_url',10,2);
function wpse312975_check_unique_url ($maybe_empty, $postarr) {

  // extract custom field from $postarr, check uniqueness

  if ($unique) return false else return true;
  }

Note: the function must return ‘true’ to halt the saving process.

If the custom field is not unique you may also want to echo a warning.