Change the output for [gallery] shortcode

The source for the gallery shortcode is in wp-includes/media.php. There are a couple of hooks in there that might work for you. Without knowing exactly what you want to do, it is impossible to be more specific.

If that doesn’t work then you can remove the shortcode:

remove_shortcode('gallery');

And add another gallery shortcode with the alterations you want. Just copy the function to functions.php or to a plugin file, rename it, edit it, and add it back with add_shortcode('gallery', 'whatever_you_named_your_function');