Don’t publish/update a post when a plugin finds errors in custom fields

welcome to WordPress Answers. I’m afraid there isn’t a (server-side) way of preventing posts being submitted.

The hook that is fired when a post is saved, save_post, is fired after the post and its custom fields are saved to the database.

However, you can always retrieve the custom field values, validate them and then delete any invalid fields

(Keep in mind that you’ll only want to delete custom fields with keys that your plug-in deals with. Users will not be impressed if they can’t save their own custom field data because your plug-in thinks its invalid. On a related note, you should prefix your custom field keys with a prefix unique to your plugin)

Anyway, do to the above… (note: this is untested)

add_action('save_post', 'my_save_post');
function my_save_post($post_id) {

    //Make sure you check this isn't an autosave.
    if ( defined( 'DOING_AUTOSAVE' ) && DOING_AUTOSAVE )
         return;

    // Check if the user can edit this post
     if ( !current_user_can( 'edit_post', $post->ID ) )
         return;

    //The keys you wish to validate. For best practise, keys should be prefixed.
    $plugin_keys = array('my_plugin_some_key','my_plugin_another_key','my_plugin_don_key');

   foreach($plugin_keys as $key):
        $values = get_post_custom_values($key, $post_id);
        //validate value
        //Rather than deleting invalid data, you could use update_post_meta
        $is_invalid = false; //set to true if the value is invalid.
        if($is_invalid)
            delete_post_meta($post_id, $key);//delete invalid data  
   endforeach;
}

A few words of warning. $values is an array of values associated with key $key. It could be an array of 1, or any number. This is something you’ll need to keep in mind when validating the values.

Secondly, delete_post_meta($post_id, $key) delete’s all of the posts custom field values with key $key. If you only wish to delete a value-key pair, you’ll want delete_post_meta($post_id, $key,$old_value).

For more information on these see the Codex on delete_post_meta and get_post_custom_values.

Finally if you wish to change the post status back to draft if you find invalid data, you may wish to look at my answer to this question.