pre_get_posts having no effect on my category archive

You’ve got a typo there, should be: is_admin(). You should enable debugging by setting define(‘WP_DEBUG’, true); in wp-config.php to see such errors.
Also there’s no need of returning anything in action hook.

add_action( 'pre_get_posts','filter_archive', 1 );
function filter_archive( $query ) {
    if ( ! is_admin() && is_category('76') && $query->is_main_query() )
        $query->set('posts_per_page', '12');
}

Sidenote: personally I prefer to rely on slugs than on IDs in such cases. If the website needs to be set up from scratch on new database, I have more flexibility. I also avoid inconsitencies between development instance and production instance which have (and should have) different databases.