get_adjacent_posts() has several filters for components of generated SQL query, including:
get_{$adjacent}_post_joinget_{$adjacent}_post_where
Where $adjacent can be next or previous. Source.
So there is no easy way to flip it from categories to tags, but you can manipulate SQL query and add own conditions (or change to them completely).