How to display posts under a specific subcategory of a given category

It should be enough to do this:

query_posts( 'posts_per_page=10&paged='.$paged.'&post_type=publication&publication_category='.$subcat );

If you look at your original code you can see why you went wrong by looking at what you are literally asking in your arguments:

I would like posts that are in the main category but are in the child category.

When what you want is:

I would like posts that are in the child category

Also, make sure to use term IDs not term Names, as they’re more reliable.

Also a final note of crucial importance

You’re checking against cat, but this is a reserved WordPress name, and will only ever hold a category ID, not a publication_category ID. If you pass anything using this GET variable, WordPress will modify the query accordingly assuming its a category. This is bad news.

So you can either rename the variable used, or, use the taxonomy archives which si what your code is hinting at. e.g. example.com/publication_category/maincategory/subcategory/ and have WordPress do all this for you.