It should be enough to do this:
query_posts( 'posts_per_page=10&paged='.$paged.'&post_type=publication&publication_category='.$subcat );
If you look at your original code you can see why you went wrong by looking at what you are literally asking in your arguments:
I would like posts that are in the main category but are in the child category.
When what you want is:
I would like posts that are in the child category
Also, make sure to use term IDs not term Names, as they’re more reliable.
Also a final note of crucial importance
You’re checking against cat, but this is a reserved WordPress name, and will only ever hold a category ID, not a publication_category ID. If you pass anything using this GET variable, WordPress will modify the query accordingly assuming its a category. This is bad news.
So you can either rename the variable used, or, use the taxonomy archives which si what your code is hinting at. e.g. example.com/publication_category/maincategory/subcategory/ and have WordPress do all this for you.