This is controlled by the wp_nav_menu argument fallback_cb, which is the wp_page_menu function by default. Set it to false to show nothing in the case a menu isn’t assigned.
wp_nav_menu( array(
'theme_location' => 'footer-menu',
'menu_class' => 'footer-links-menu',
'fallback_cb' => false,
) );