How to get “last year from [spec year] that has posts”?

You could do a query for a single post made before the currently viewed year. When sorted by date in descending order (the default) the first result will be from the most recent year with posts.

$current_year = get_query_var( 'year' );

$previous_posts = get_posts( array(
    'numberposts' => 1,
    'date_query'  => array(
        'year'    => $current_year,
        'compare' => '<',
    ),
) );

if ( ! empty( $previous_posts ) ) {
    $previous_year = get_the_time( 'Y', $previous_posts[0] );
}

You can do the same to get the next year with posts, but you need to swap the order, otherwise you’ll get the year of the latest post:

$next_posts = get_posts( array(
    'numberposts' => 1,
    'order'       => 'ASC',
    'date_query'  => array(
        'year'    => $current_year,
        'compare' => '>',
    ),
) );

if ( ! empty( $next_posts ) ) {
    $next_year = get_the_time( 'Y', $next_posts[0] );
}