You could try putting this code in your functions.php
function remove_contact_nav( $nav_menu, $args ){
if( is_page_template('template-contact.php') || is_page( 'contact' ) ) {
$nav_menu = null;
}
return $nav_menu;
}
add_filter( 'wp_nav_menu', 'remove_contact_nav', 11, 2 );
The if condition need to be modified as per your template name or the second condition would also do if your contact page slug is contact