Is the only way to add taxonomy terms via an admin panel?

Question

I think you are looking for wp_insert_term().

An example from the Codex:

$parent_term = term_exists( 'fruits', 'product' ); // array is returned if taxonomy is given
$parent_term_id = $parent_term['term_id']; // get numeric term id
wp_insert_term(
  'Apple', // the term 
  'product', // the taxonomy
  array(
    'description'=> 'A yummy apple.',
    'slug' => 'apple',
    'parent'=> $parent_term_id
  )
);