Filter for replacing the WP_query object for a given category

You’re looking for template_redirect:

add_action('template_redirect', 'my_category_template');

function my_category_template(){
  if(is_category(5)){ // <- category ID/title/slug
     require "path/to/your/template.php";
  }
}

If you only need to change the query arguments, then see Mike’s answer to this question