Without knowing what your post_gallery filter function does, there’s no way to give a proper answer.
However, if you are using the post_gallery filter, and returning new markup, then yes, it will be used in feeds too. The code you mention commenting out happens after the post_gallery filter, and won’t get executed at all if you’re returning a different output from that filter, as per the following code:
// Allow plugins/themes to override the default gallery template.
$output = apply_filters('post_gallery', '', $attr);
if ( $output != '' )
return $output;
If the post_gallery filter returns anything at all, then that is what is output, period. The if is_feed code you mention will not apply since it never makes it that far.
Examine your post_gallery hooked function. The problem probably lies there somewhere.
Edit: Found the link to your code at the bottom of your post. It was sorta hidden.
You have two major problems. Firstly, you had the if is_feed code inside your own function, and commenting it out was indeed the correct thing to do. Your function is supposed to produce your output. If you’re producing different output, then you need to change that, obviously.
Second, this is incorrect:
add_shortcode('post_gallery', 'flipboard_gallery_shortcode', 10, 2);
The post_gallery hook is a filter hook, not a shortcode. Change that line to this:
add_filter('post_gallery', 'flipboard_gallery_shortcode', 10, 2);