Several things to consider so let’s look at each one individually:
Preserve Admin Screens
As written, your function affects every page for that post-type, including your admin screens which will hide your filtered posts from all post listings.
Test to see if an admin screen has fired the post type query with is_admin() and return early if so. Your function is hooked to pre_get_posts and will fire several times on every page load so the concept of returning early and avoiding additional code is important to avoid performance problems.
Post Type Archive
Your use of is_archive() is not doing what you expect it to. There are no parameters available for is_archive() so the my-post-type is ignored. https://developer.wordpress.org/reference/functions/is_archive/
Using is_post_type_archive( 'my-post-type' ) is the correct way to test if the query is your CPT archive page. Return early if the query is not (!).
https://developer.wordpress.org/reference/functions/is_post_type_archive/
Exclude Taxonomy
Adding a tax query argument to the main query using the “NOT EXISTS” operator will eliminate any post that has any term for your taxomomy. NOTE: Taxonomy query arguments are an array of array by design.
Tax query documentation: https://developer.wordpress.org/reference/classes/wp_query/#taxonomy-parameters
Don’t forget to update my-post-type and my-taxonomy with your actual slugs.
Putting it all together we get:
function wpse_exclude_mytaxonomy_from_archive( $query ) {
// Three conditions that we want to exit early for
if ( ( is_admin() ) || ( ! is_post_type_archive( 'my-post-type' ) ) || ( ! $query->is_main_query() ) ) {
return;
}
// Modify the query to exclude posts with any My Taxonomy term
$taxquery = array(
array(
'taxonomy' => 'my-taxonomy',
'operator' => 'NOT EXISTS',
)
);
$query->set( 'tax_query', $taxquery );
}
add_action('pre_get_posts', 'wpse_exclude_mytaxonomy_from_archive');