Since your first get_categories call will load all terms into memory, you can just loop over ’em and check if they have an ancestor with ID 55, and WordPress won’t ever hit the db again:

$terms = get_terms([
     'taxonomy'   => 'category',
     'hide_empty' => false,
]);

foreach ( $terms as $k => $term ) {
    if ( in_array( 55, get_ancestors( $term->term_id, 'category', 'taxonomy' ) ) )
        unset( $terms[ $k ] );
}