How to ensure buddypress custome profile field is unique

You should be able to enforce uniqueness with a filter something like this (untested…consider this pseudocode, just the basic idea):

add_filter( 'bp_xprofile_field_type_is_valid', function( $validated, $values, $field ) {
    global $bp, $wpdb;

    // you'll need to constrain this to your particular field type
    if ( $field->id === $your_field_id ) {
        $results = $wpdb->get_results( $wpdb->prepare(
            sprintf( "SELECT COUNT(*) FROM {$bp->profile->table_name_data} WHERE field_id = %s AND value="%s"", $field->id, (string) $values )
        ) );
        $validated = ( count( $results ) === 0 );
    }

    return $validated;
}, 10, 3 );

I’m not aware of any documentation about this outside the code itself unfortunately. But take a look at the field type classes for xprofile – the code is pretty readable and there are decent comments as well.