How to use wp_nav_menu with hashtag links?

You already hinted it yourself: When you look at wp_get_nav_menu_items() you’ll see that the resulting Array of items gets mapped over with wp_setup_nav_menu_item() – in other words, this function gets applied to each and every nav menu item.

In there you can hook into the 'wp_setup_nav_menu_item'-filter with its $menu_item argument, or use one of the filters in the applied functions in between. The level of control just lays in what you use where (look up source). The argument itself is an object that you can influence.