WordPress custom archive page

Of course it breaks. Pagination depends on the main query saved in the $wp_query object and you have overwritten that. When the next page load the ordinary wp_query that you have not overwritten will run and it won’t match the one you’ve altered.

Create a filter for pre_get_posts to alter the main query.

function pregp_wpse_97354($qry) {
  // control where this runs
  if (is_main_query() && $qry->is_archive()) {
    $qry->set('offset',5);
  }
}
add_action('pre_get_posts','pregp_wpse_97354');

Notice // control where this runs. The line immediately following determines where this filter runs. I’d don’t enough about your setup to get that right so I made something up. I am sure that that needs to be more specific.