Well, I found not an answer, but a workaround for this question. In the_mx_limited_gallery function, I added a negated is_single check around most of the contents of the function, so basically the limited posts and custom code show only if you are on a page other than a single post.
This isn’t how I originally intended, but it works for the time being. Below is the adjusted code:
function the_mx_limited_gallery( $attr ) {
$post = get_post();
$attachment_ids = the_mx_get_limited_gallery_ids();
$link_image_to = the_mx_medialink_switcher(); // Customizer controls
$mx_colcount = the_mx_gal_colcount_switcher(); // Customizer controls
if( get_post_format() == 'gallery' ) { // opens gallery post format check
if( !is_single() ) { // opens non single page if statement
// setup shortcode attributes
$atts = shortcode_atts( array(
...
} // closes non single page if statement
} // closes gallery post format
}
add_filter( 'post_gallery', 'the_mx_limited_gallery', 10, 1 );
I forgot the add_filter part in the original question.
If anyone still can find a solution that only adjusts the shortcode attributes, I am still open for suggestions or answers. For now, I will mark this as answered.