Add the current menu item CSS class to a custom page type archive in WordPress menu

This is what you want to do,

  add_filter( 'wp_get_nav_menu_items', 'cpt_archive_menu_filter', 10, 3 );
  function cpt_archive_menu_filter( $items, $menu, $args ) {
    foreach ( $items as &$item ) {
    if ( $item->type != 'custom' ) continue;
      if (  get_query_var( 'post_type' ) == 'your-post-type' && $item->title == 'Title of Link' ) {
        $item->classes []= 'class-name-your-want';
        } 
    }
    return $items;
  } 

Although the above will work, Custom Links get assigned the current-menu-item class so this shouldn’t be necessary, if I follow you correctly.