Limit upload by file type only for certain custom post type

function wpse_59621_mimes_filter( $mimes ) {
    return array( 'pdf' => 'application/pdf' );
}

function wpse_59621_delay_mimes_filter( $value ) {
    if ( isset( $_REQUEST['post_id'] ) && get_post_type( $_REQUEST['post_id'] ) === 'my_post_type' )
        add_filter( 'upload_mimes', 'wpse_59621_mimes_filter' );
    else
        remove_filter( 'upload_mimes', 'wpse_59621_mimes_filter' );

    return $value;
}

add_filter( 'wp_handle_upload_prefilter', 'wpse_59621_delay_mimes_filter' );

Let us know how it goes – this is untested, but I’m confident!