You could do a query for a single post made before the currently viewed year. When sorted by date in descending order (the default) the first result will be from the most recent year with posts.
$current_year = get_query_var( 'year' );
$previous_posts = get_posts( array(
'numberposts' => 1,
'date_query' => array(
'year' => $current_year,
'compare' => '<',
),
) );
if ( ! empty( $previous_posts ) ) {
$previous_year = get_the_time( 'Y', $previous_posts[0] );
}
You can do the same to get the next year with posts, but you need to swap the order, otherwise you’ll get the year of the latest post:
$next_posts = get_posts( array(
'numberposts' => 1,
'order' => 'ASC',
'date_query' => array(
'year' => $current_year,
'compare' => '>',
),
) );
if ( ! empty( $next_posts ) ) {
$next_year = get_the_time( 'Y', $next_posts[0] );
}