Yes use wp_query 🙂
Anyway, to prevent loading posts while you’re out of pages, you need to know the current page number last loaded and the total number of pages that exist.
Total pages that exist: $max = $wp_query->max_num_pages;
Current page: just count it in your load loop, so set the var to 1, then iterate it each time the load button is clicked.
Then you can just compare the 2 when the button is clicked and before you do the operation:
if(pageNum <= max) { //do your stuff }
An excellent tut on this is here, I used it to build my own stuff: http://www.problogdesign.com/wordpress/load-next-wordpress-posts-with-ajax/
Concerning your 2nd non-related question: it’s probably because of your markup. I see you load boxes in a wrapper ‘work-items’ but the new boxes added after load don’t go in that wrapper. :nth only works when it has a clear parent, so #work-items .box:nth(4n)
Good luck!