The way you do it is fine.
Also have a look at how Tom McFarlins WordPress Plugin Boilerplate does it:
if ( is_admin() && ( ! defined( 'DOING_AJAX' ) || ! DOING_AJAX ) ) {
require_once( plugin_dir_path( __FILE__ ) . 'admin/class-plugin-name-admin.php' );
add_action( 'plugins_loaded', array( 'Plugin_Name_Admin', 'get_instance' ) );
}