failed to filter hook `get_terms_defaults`

Filters are for modifying data that’s passed through them. You need to take the input, modify it, then return it.

function exclude_not_allowed_terms( $args, $taxonomies ) {
    if(!current_user_can('edit_others_posts')
      && in_array( 'my-taxonomy', $taxonomies ) ) {
        $args['meta_query'] = array(
            array(
                'key'    => 'owner',
                'value'  => get_current_user_id()
            )
        );
    }
    return $args;
}
add_filter( 'get_terms_defaults','exclude_not_allowed_terms', 20, 2 );