What your asking is if WordPress is modular. Almost like many plugins making up the core and you could use one of those plugins. There are some files with classes you can call on their own but even then some of the methods use even the smallest of core WP functions.
I wouldn’t worry about load time. Maybe 5 years ago but if your site needs a CMS driving it then WordPress will be more of a benefit than a hindrance.
This page explains how to load WordPress externally i.e. in phpBB or a basic website…
http://codex.wordpress.org/Integrating_Wordpress_with_Your_Website
It is all about these lines…
<?php
/* Short and sweet */
define('WP_USE_THEMES', false);
require('./wp-blog-header.php');
?>
The WP_USE_THEMES set to false avoids loading some stuff but apart from that there isn’t option to exclude more of the core.