The number of posts attached can be found here: $term->count
.
So you could do the following:
$class_empty_term = '';
if( $term->count == 0 )
$class_empty_term = 'empty';
$term_list .= '<div class="' . $class_empty_term . ' grid__item ...">';
$term_list .= '<div class="panel text--center">';