try to use undocumented filter pre_insert_term.
What you need is not a filter， but an action called before insert terms, i have not found WordPress have this kind of action.
for another way of thinking, you can use create_term or created_term action, although they are called after term created, if a user have no permission, we can delete the term they just created.