You need to change your 'orderby'
parameter.
$term_args = array(
'number' => $per_page,
'offset' => $offset,
'orderby' => 'slug',
);
$terms = get_terms('my_taxonomy', $term_args);
WordPress slugs automatically remove punctuation and such so this is probably all you need to do in order to accomplish what you want.