This is what you want to do,
add_filter( 'wp_get_nav_menu_items', 'cpt_archive_menu_filter', 10, 3 );
function cpt_archive_menu_filter( $items, $menu, $args ) {
foreach ( $items as &$item ) {
if ( $item->type != 'custom' ) continue;
if ( get_query_var( 'post_type' ) == 'your-post-type' && $item->title == 'Title of Link' ) {
$item->classes []= 'class-name-your-want';
}
}
return $items;
}
Although the above will work, Custom Links get assigned the current-menu-item
class so this shouldn’t be necessary, if I follow you correctly.