Fix custom query pagination without changing site-wide posts-per-page settings

I assume you have a custom post type: publication

in functions.php place that code:

function mk_parse_query($query) {
  if ($query->query_vars['post_type'] == 'publication')) {
        set_query_var( 'posts_per_page', 5 );
  }
}
if( !is_admin() ) {
    add_filter('parse_query','mk_parse_query');
}

Similar code works for me, but in my case I was struggling with 404 on custom taxonomies paged urls.

Hope this helps.