How to check for duplicate record before saving a Custom Post Type

See this answer for solution, see comments there about error messages. That should work with Classic editor.

It could work with Gutenberg too, but it would need additional effort to get correct error messages.

UPDATE:

If using classic editor and preventing publishing of invalid records is enough, you can use this code:

function my_save_post($post_id) {

    if (.. record is not valid ..) {
        // remove Post Published message
        add_filter( 'redirect_post_location', function ( $location ) {
            return preg_replace( '/&message=\d+/', '', $location );
        } );

        // this is important to prevent recursive loop
        remove_action( 'save_post', 'my_save_post' );
        wp_update_post( [ 'ID' => $post_id, 'post_status' => 'draft' ] );
        add_action( 'save_post', 'my_save_post' );

        // admin_notice will not work because of redirect
        // try to change message in redirect_post_location - see above
        // or store transient and here and use it on next request
        // to show error message
        set_transient( 'my_save_errors_' . $post_id, $error_message, 20 );
    }
}

add_action( 'save_post', 'my_save_post' );