I assume you have a custom post type: publication
in functions.php place that code:
function mk_parse_query($query) {
if ($query->query_vars['post_type'] == 'publication')) {
set_query_var( 'posts_per_page', 5 );
}
}
if( !is_admin() ) {
add_filter('parse_query','mk_parse_query');
}
Similar code works for me, but in my case I was struggling with 404 on custom taxonomies paged urls.
Hope this helps.