Show one post per author and order by custom field

You use a AND statement after a GROUP BY statement. That wont work.

Try altering the WHERE statement (posts_where_paged filter) to add the AND pm.meta_key = 'KEY' line.

Additionally you can use the posts_orderby filter to alter the ORDER BY.

EDIT

You can find some additional information here.

tech