Ignore punctuation marks in taxonomy order by name

You need to change your 'orderby' parameter.

$term_args = array(
    'number' => $per_page,
    'offset' => $offset,
    'orderby' => 'slug',
);
$terms = get_terms('my_taxonomy', $term_args);

WordPress slugs automatically remove punctuation and such so this is probably all you need to do in order to accomplish what you want.