wp_nav_menu always falls back to a menu

wp_nav_menu() indeed tries a lot to provide you with a menu, and fallback_cb is only executed when nothing else works. From the code:

• If menu is provided and refers to an existing menu (looked up via wp_get_nav_menu_object(), which accepts an id, slug or name), this will be the menu
• Otherwise, if theme_location is set to a registered menu location, this will be passed to wp_get_nav_menu_object()
• Otherwise, WordPress will search for the first existing menu that has items and use that
• Otherwise, fallback_cb is called, which by default is wp_page_menu, which is a menu of all the pages

So if you only want to use the menu argument, you should test this yourself by calling wp_get_nav_menu_object(). Only if this returns something you should call wp_nav_menu().