Show archives by year from just one category

If I understand you question clearly that we just need to get post in that submenus from only cat id = 4 , this will make your archive.php to work only for cat id = 4

Please add below code to functions.php:

function wpse75668_filter_pre_get_posts( $query ) {


   if ( $query->is_main_query() && ! is_admin() ) {

      if ( is_date() ) {

        $query->set( 'cat', '4' );
     }
   }
 }
 add_action( 'pre_get_posts', 'wpse75668_filter_pre_get_posts' );

Thanks!