You can check post data in $_POST array on save_post_{custom_post_type_name} action hook.
And if data is not appropriate or according to server side validation and redirect page with custom error args using wp_redirect and , you can grab the errors and display admin notices using admin_notice.