Assuming you are trying to remove the default nav_menus panel, you have the id wrong. Also, you’ll need to add a priority of at least 20 to the customize_register hook, assuming you’re using that hook.
function remove_customizer_settings( $wp_customize ){
$wp_customize->remove_panel('nav_menus');
}
add_action( 'customize_register', 'remove_customizer_settings', 20 );