Get next and previous 3 posts in a term in single post page

You second approach (EDIT 2) is quite buggy and inefficient unfortunately. Also, you are not going to do this in one query.

As I already stated, you need to look at the approach in this answer I have recently done. You were almost there in your first edit, the only problem is, you cannot do this in one query, you will have to do two, one to get the previous set of posts, the other to get the next set of posts.

I have optimized the code to do only get the necessary info and nothing more. WP_Query results is also cached, so you really don’t have to to worry that much about efficiency.

I’m not going to rerun through all the detail again in this answer, you should go through the linked post in detail, but you can try something like this (CAVEAT: Untested. Please see the notes in linked answer)

$post_object = get_queried_object();
$terms = wp_get_post_terms( $post_object->ID, 'series', array( 'fields' => 'ids' ) ); // Set fields to get only term ID's to make this more effient
$args = [
    'post_type' => $post_object->post_type,
    'tax_query' => [
       [
        'taxonomy' => 'series',
        'terms'    => $terms,
      ]
    ],
    'posts_per_page' => 3,
    'order' => 'ASC' // CHANDE TO DESC IF NOT CORRECT
    /* make query more efficient */
    'no_found_rows' => true,
    /* dont let filters modify query */
    'suppress_filters' => true,
    'date_query' => [
      [
        'before'    => $post_object->post_date,
        'inclusive' => false
      ],
    ]
];
$q1 = new WP_Query( $args );
var_dump( $q1->posts );

$args1 = [
    'post_type' => $post_object->post_type,
    'tax_query' => [
       [
        'taxonomy' => 'series',
        'terms'    => $terms,
      ]
    ],
    'posts_per_page' => 3,
    'order' => 'DESC' // CHANDE TO ASC IF NOT CORRECT
    /* make query more efficient */
    'no_found_rows' => true,
    /* dont let filters modify query */
    'suppress_filters' => true,
    'date_query' => [
      [
        'after'    => $post_object->post_date,
        'inclusive' => false
      ],
    ]
];
$q2 = new WP_Query( $args1 );
var_dump( $q2->posts );