It seems that you are looking for a plugin that applies masonry layout on your photos.
When selecting appropriate WP plugin keep in mind that you presumably want one that gives you whole gallery functionality (lightbox, lazy-load or pagination etc.)
Free version of this plugin seems to have at least some of these.
As for jQuery plugin, if your project is not commercial (or if it is and you don’t mind spending a few dollars), I recommend using Isotope. It’s as simple to work with as using jQuery function on a container of your photos. It is also offering nice filtering features, fancy effects and full responsiveness.