check out this thread – I’m afraid nothing has changed since then, there is still no way of filtering the description field (it’s just html hardcoded in the file https://github.com/WordPress/WordPress/blob/master/wp-admin/edit-tags.php#L484, so you can’t remove it with php without editing the core files, which is never a right way to go).
The hook you’re using, {$taxonomy}_edit_form_fields is fired on single term edit screen before the standard fields are printed, so you can use it to add something more, but not for filtering standard fields.
I’d say you need to hold on to your JS solution for now or even better – go with CSS display: none; solution to make sure the field doesn’t show up when JavaScript is disabled and to avoid flickering, like mentioned here.