The simplest method may be to just save titles as last name / first name, then they will naturally sort. You can then reverse the title when you output it in the template.
Your attempt to modify the orderby query var is fundamentally flawed. Order is created by MySQL, which doesn’t understand PHP, and can’t be modified after the query has happened.
If the results are ever to be paginated, the order has to be done via the query, which will require modification of the raw SQL sent to the database via filters.
If the results are not paginated, you can sort the posts manually with PHP’s usort:
$loop = new WP_Query( array(
'posts_per_page' => -1,
'post_type' => 'actors'
) );
// sort function
function wpd_last_name_sort( $a, $b ) {
$a_last = end(explode(' ', $a->post_title));
$b_last = end(explode(' ', $b->post_title));
return strcasecmp( $a_last, $b_last );
}
// sort posts
usort( $loop->posts, 'wpd_last_name_sort' );
// set post object to first post in sorted list
$loop->post = $loop->posts[0];
// run the loop...
You probably also want to consider the case where names are more than just two words. I would personally make the titles last name, first name and let MySQL do the work.