How to filter a dd/mm/yyyy date from a custom field in a query

The filter on posts_orderby should return a string that does not begin with ORDER BY, WordPress will add that itself.

By default WordPress will filter by post_date. If you don’t want that, you should overwrite the order clause, not append to it. So your filter should look like this:

function filter_orderby( $orderby ) {  
    global $wpdb;
    return " STR_TO_DATE({$wpdb->postmeta}.meta_value, '%d/%m/%Y') ASC ";
}