add_query_arg() and empty variables inside

You can instantiate your array before-hand, optionally populate it, and pass it to the add_query_arg() function like so:

$url_query_args = array();

if( isset( $search ) ) {
    $url_query_args['search'] = $search;
}

if( isset( $category ) ) {
    $url_query_args['category'] = $category;
}

if( isset( $filter ) ) {
    $url_query_args['filter'] = $filter;
}

esc_url( add_query_arg( $url_query_args, '/site' ) );

Or you could loop through an array of possible query args:

$url_query_args = array();
$possible_args  = array(
    'search',
    'category',
    'filter',
);

foreach( $possible_args as $arg ) {

    if( ! isset( ${$arg} ) ) {
        continue;
    }

    $url_query_args[ $arg ] = ${$arg};

}

esc_url( add_query_arg( $url_query_args, '/site' ) );