Get all terms linked to a post and include those who not connected from get_terms and sort

You just need to get the term id’s of the terms assigned to a post, and then exclude those terms from get_terms()

You can try the following (Just note, the syntax for get_terms() changed in v4.5, I’ll be using the new syntax)

$args       = [];
$taxonomy   = 'my_tax';
$post_terms = get_the_terms( get_the_ID(), $taxonomy ); // Use wp_get_post_terms() if you need special ordering
// Only display the post terms and exclude them if there are terms and no WP_Error
if (    $post_terms
     && !is_wp_error( $post_terms )
) {
    $term_ids = [];
    // Display the list of post terms
    foreach ( $post_terms as $term_object ) {
        // Display term data 

        // Build an array of term ids
        $term_ids[] = $term_object->term_id;
    }

    // Set these id's to the exclude parameter
    $args['exclude'] = $term_ids;
}

// Now we can get the list of all term
$args['taxonomy'] = $taxonomy;
$terms            = get_terms( $args );
// Check if we have terms and no WP_Error
if (    $terms
     && !is_wp_error( $terms )
) {
    // Display the terms
    foreach ( $terms as $term ) {
        // Output the terms as needed
    }
}