Calling the first & last post by category in custom post type

It is sometimes irritating that some build-in functions don’t have appropriate filters to modify their output. get_boundary_post() is one of those build-in functions. Unfortunately it does not get posts according to post type, only taxonomy. You can still however make use of this function to get the first and last post. It does support the $in_same_term parameter, the same as the next_post_link() and previous_post_link() functions.

Just a note here, the codex page for get_boundary_post() is incorrect, there are 4 parameters, like below (Check the source, currently lines 1694 – 1750 in wp-includes/link-template.php)

get_boundary_post( $in_same_term = false, $excluded_terms="", $start = true, $taxonomy = 'category' )

If you need more control over this function, simply copy the complete function to functions.php, rename it and make the desired modifications

Use get_boundary_post() as follows to get the first and last post in the same category from a taxonomy called mytax

//First post
$first = get_boundary_post( true, '', true, 'mytax' );
echo apply_filters( 'the_title', $first[0]->post_title );
//Last post
$last = get_boundary_post(true, '', false, 'mytax');
echo apply_filters( 'the_title', $last[0]->post_title );

Just one note, if you set the $in_same_term parameter to true in either post links, you need to set the taxonomy as well if the taxonomy is not the default category taxonomy, for example

previous_post_link( '%link', 'Previous post in same term', TRUE, ' ', 'mytax' );

EDIT

I think most probably why you get the same post returned is the fact that get_boundary_post uses get_posts which uses WP_Query which default to the post type post. As I said, there is no filter here.

The best will be to copy the function, rename it and modify as needed.

Try something like this

function get_my_custom_boundary_post( $in_same_term = false, $excluded_terms="", $start = true, $taxonomy = 'category' ) {
    $post = get_post();
    if ( ! $post || ! is_single() || is_attachment() || ! taxonomy_exists( $taxonomy ) )
        return null;

    $query_args = array(
        'post_type' => 'MYPOSTTYPE',
        'posts_per_page' => 1,
        'order' => $start ? 'ASC' : 'DESC',
        'update_post_term_cache' => false,
        'update_post_meta_cache' => false
    );

    $term_array = array();

    if ( ! is_array( $excluded_terms ) ) {
        if ( ! empty( $excluded_terms ) )
            $excluded_terms = explode( ',', $excluded_terms );
        else
            $excluded_terms = array();
    }

    if ( $in_same_term || ! empty( $excluded_terms ) ) {
        if ( $in_same_term )
            $term_array = wp_get_object_terms( $post->ID, $taxonomy, array( 'fields' => 'ids' ) );

        if ( ! empty( $excluded_terms ) ) {
            $excluded_terms = array_map( 'intval', $excluded_terms );
            $excluded_terms = array_diff( $excluded_terms, $term_array );

            $inverse_terms = array();
            foreach ( $excluded_terms as $excluded_term )
                $inverse_terms[] = $excluded_term * -1;
            $excluded_terms = $inverse_terms;
        }

        $query_args[ 'tax_query' ] = array( array(
            'taxonomy' => $taxonomy,
            'terms' => array_merge( $term_array, $excluded_terms )
        ) );
    }

    return get_posts( $query_args );
}

Then just change MYPOSTTYPE to your actual post type, and also change the previously given code to

//First post
$first = get_my_custom_boundary_post( true, '', true, 'mytax' );
echo apply_filters( 'the_title', $first[0]->post_title );
//Last post
$last = get_my_custom_boundary_post(true, '', false, 'mytax');
echo apply_filters( 'the_title', $last[0]->post_title );