How to order posts tag by tag?

Like Eugene mentioned in his answer you need to run a query for each tag. I would create a foreach loop that went through each tag then queried the latest 2 posts from each.

$tags = get_tags();
foreach ( $tags as $tag ) {

    echo '<h3>' .$tag->name. '</h3>';
    $tag_query = new WP_Query( array( 
                              'tag_id' => $tag->term_id,
                              'posts_per_page' => 2,
                              'no_found_rows' => true,
                               ) );
        while ( $tag_query->have_posts() ) : $tag_query->the_post();
        // Do stuff
        endwhile; wp_reset_postdata();
    }