Dunno if you looked at add_user_meta, but it takes an optional fourth parameter.
function add_user_meta( $user_id, $meta_key, $meta_value, $unique = false )
Send “true” to the $unique value and it won’t insert it twice.
https://developer.wordpress.org/reference/functions/add_user_meta/