How to validate custom fields in custom post type?

You’re on the right track. I test the fields in the save_post callback, and then use admin notices to display errors to the user when a field fails validation. They show up just in a highlighted box at the top of the page, just like any errors/messages that WordPress itself generates.

Here’s a simple example of creating an admin notice:

function my_admin_notice()
{
    ?>

    <div class="updated">
       <p>Aenean eros ante, porta commodo lacinia.</p>
    </div>

    <?php
}
add_action( 'admin_notices', 'my_admin_notice' );

That’s not very practical, though. In a situation like this, you really just want a function that you can pass a message to. Something like,

if( $pizza != 'warm' )
    $notices->enqueue( 'Pizza is not warm', 'error' );

So, you can write that enqueue() function yourself (along with a function to print the notices), or you can include a library like IDAdminNotices.

Here’s an example from a plugin I wrote. This uses notice enqueue/print functions that are built into the class itself, rather than including an external library.

public function saveCustomFields( $postID )
{
    // ...

    if( filter_var( $_POST[ self::PREFIX . 'zIndex'], FILTER_VALIDATE_INT ) === FALSE )
    {
        update_post_meta( $post->ID, self::PREFIX . 'zIndex', 0 );
        $this->enqueueMessage( 'The stacking order has to be an integer.', 'error' );
    }   
    else
        update_post_meta( $post->ID, self::PREFIX . 'zIndex', $_POST[ self::PREFIX . 'zIndex'] );

    // ...
}
add_action( 'save_post',    array( $this, 'saveCustomFields' );