filter “inactive” categories from wp_list_categories?

If you are talking 5 or 10 categories then just save them as a comma-separated string of IDs in using update_option() and use it as you suggested with your wp_list_categories() 'exclude' argument. And this answer should show you how to add a field to the category screen: