Would there be anything stopping me from removing both wp_head and wp_footer?

Don’t remove wp_head() or wp_footer() as they are required for triggering other hooks. Instead, conditionally remove scripts / stylesheets based on page type, page ID, page template, post category, etc.

I reckon you mostly need to worry about this on the site’s front page, since all your site CSS and main JavaScript will be loaded on that page and cached for the rest of the website (yes?) and once people have downloaded everything on the front page and cached various assets, any gzipped scripts and stylesheets should be pretty quick to load on whatever page follows.

Here’s a rip from a site I did last year, where I knocked out a bunch of scripts and CSS just from the front page.

// actions to remove unnecessary scripts and styles
add_action('wp_print_scripts', 'wpse_84052_removeScripts', 100);
add_action('wp_print_styles', 'wpse_84052_removeStyles');

/**
* performance: remove some scripts we don't need on splash page
*/
function wpse_84052_removeScripts() {
    if (is_front_page()) {
        // events manager
        wp_dequeue_script('events-manager');
        wp_dequeue_script('events-manager-pro');

        wpse_84052_removeObjectFilters('wp_head', 'EM_Pro');
        wpse_84052_removeObjectFilters('wp_head', 'EM_Coupons');

        // NextGEN Gallery
        wp_dequeue_script('ngg-slideshow');
        wp_dequeue_script('shutter');
    }
}

/**
* performance: remove some stylesheets we don't need on splash page
*/
function wpse_84052_removeStyles() {
    if (is_front_page()) {
        // WP Flexible Map plugin styles
        wp_dequeue_style('flxmap');

        // NextGEN Gallery plugin styles
        wp_dequeue_style('NextGEN');
        wp_dequeue_style('shutter');

        // Events Manager plugin styles
        wp_dequeue_style('events-manager');

        // wp-category-posts-list plugin styles
        wp_dequeue_style('wp_cpl_css_3');
        wp_dequeue_style('wp_cpl_css_2');
        wp_dequeue_style('wp_cpl_css_1');
        wp_dequeue_style('wp_cpl_css_0');
        wp_dequeue_style('wp-cpl-base-css');
    }
}

/**
* remove filters that are methods of an object of some class
* @param string $filterName name of action or filter hook
* @param string $className name of class for object method
*/
function wpse_84052_removeObjectFilters($filterName, $className) {
    global $wp_filter;

    // must take a variable to iterate over array of filters,
    // else a subtle reference bug messes up the original array!
    $filters = $wp_filter[$filterName];

    foreach ($filters as $priority => $hooks) {
        foreach ($hooks as $idx => $filter) {
            // check for function being a method on a $className object
            if (is_array($filter['function']) && (is_a($filter['function'][0], $className) || $filter['function'][0] === $className)) {
                remove_filter($filterName, $idx, $priority);
                break;
            }
        }
    }
}