I don’t like recommending a plugin, but this plugin does exactly what you need, and it is the easiest way to go.
Just before I start, I would like to state that I don’t have any affiliation regarding the plugin or its authors, nor am I marketing or trying to market the plugin in any way.
OK, that out of the way, you should download and install the Simple Yearly Archive plugin. I quote from the plugin page
Simple Yearly Archive is a rather neat and simple WordPress plugin that allows you to display your archives in a year-based list. It works mostly like the usual WP archive, but displays all published posts seperated by their year of publication. That said, it’s also possible to restrict the output to certain categories, and much more
The how-to-use page can be found here.