How to remove feeds from WordPress totally?

First step: remove the feed links from the section of your site.

<?php
add_action( 'wp_head', 'wpse33072_wp_head', 1 );
/**
 * Remove feed links from wp_head
 */
function wpse33072_wp_head()
{
    remove_action( 'wp_head', 'feed_links', 2 );
    remove_action( 'wp_head', 'feed_links_extra', 3 );
}

Next up, let’s remove the feed endpoints from WP. Hook into init, globalize $wp_rewrite then set the feeds to an empty array. This effectively stops WordPress from adding feed rewrites. It’s also super hackish and will probably break at some point in the future.

<?php
add_action( 'init', 'wpse33072_kill_feed_endpoint', 99 );
/**
 * Remove the `feed` endpoint
 */
function wpse33072_kill_feed_endpoint()
{
    // This is extremely brittle.
    // $wp_rewrite->feeds is public right now, but later versions of WP
    // might change that
    global $wp_rewrite;
    $wp_rewrite->feeds = array();
}

But, if it breaks, that’s okay, because we’ll redirect feeds to the home page.

<?php
foreach( array( 'rdf', 'rss', 'rss2', 'atom' ) as $feed )
{
    add_action( 'do_feed_' . $feed, 'wpse33072_remove_feeds', 1 );
}
unset( $feed );
/**
 * prefect actions from firing on feeds when the `do_feed` function is 
 * called
 */
function wpse33072_remove_feeds()
{
    // redirect the feeds! don't just kill them
    wp_redirect( home_url(), 302 );
    exit();
}

And the last step: an activation hook to set our rewrite feeds to an empty array and flush the rewrite rules.

<?php
register_activation_hook( __FILE__, 'wpse33072_activation' );
/**
 * Activation hook
 */
function wpse33072_activation()
{
    wpse33072_kill_feed_endpoint();
    flush_rewrite_rules();
}

All that as a plugin.