Can you add shortcode as a menu item link?

I am not sure about the Shortcode in Menu; but Menu Walker Class is your best bet to achieve what you want. Walker Class will pass on each Menu element to your code, this is where you can catch your menu item and make the desired modal pop-up link for a href of your menu item.

For more information on Walker class, see this link:
Class Reference/Walker