If you don’t use a plugin, then you’ll essentially have to write your own which strikes me as counter-productive. I would recommend using a plugin and figuring out how to make it work.
I’ve used mfields’ Taxonomy Images to great effect. The way you access the image is certainly a little funky in some situations, but it works. There are some good code snippets in the WordPress.org support forum for that plugin in addition to what is provided on the Plugin page.