Responsive WordPress Gallery

You need add some CSS to make it auto clear left after each 3th item and need to hide <br> tag

Update your code like this-

/* For displaying 3 columns on tablet */
@media only screen and (max-width: 800px) {
    .gallery-columns-5 .gallery-item {
        width: 33% !important;
    }
    .gallery-columns-5 br {
        display: none;
    }
    .gallery-columns-5 .gallery-item:nth-of-type(3n+1) {
        clear: left;
    }
}