Query Set Order By Author

To order by the authors nicename is not nativly supported by the WP_Query. If you order by author, as you have experienced, you order by the ID of the author.

So, you would need to actually go into the SQL clauses and manage this by yourself.

This is a quick and dirty hack to do so:

<?php

add_action( 'pre_get_posts', function( $query ) {

    if (
        ! $query->is_main_query() 
        || 'author' !== $query->get( 'orderby' )
    ) {
        return;
    }

    add_filter( 'posts_clauses', '177581_add_nicename_orderby', 10, 2 );
} );

function 177581_add_nicename_orderby( $clauses, $query ) {

    global $wpdb;

    $clauses['join'] = 'LEFT JOIN ' . $wpdb->users . ' AS author ON ( ' . $wpdb->posts . '.post_author = author.ID)';
    $clauses['orderby'] = 'author.user_nicename ' . $query->get( 'order' );
    return $clauses;
}

What we need to do:
In the first step, we check, if we want to order by author, what we treat equally to order by its nicename. If this is the case, we hook into the posts_clauses-filter, where we can actually alter the SQL statements.

We have to

  1. Join the user table, as we find the nicename in this table.
  2. Alter our orderby statement accordingly.