Thanks to this great tutorial, I just found the answer myself.
WordPress offers the action hooks sb_reference_category_add_form_fields, sb_reference_category_edit_form_fields, create_sb_reference_category and edited_sb_reference_category (these are named after the custom taxonomy, in my case sb_reference_category). With these hooks, it is easy as pie to add and save a custom field to the taxonomy.