Having problems loading Jquery in functions.php

You have multiple errors here.

1. You’re including CSS and JS on the same and wrong hook
2. For JS you could use wp_enqueue_scripts
3. For CSS you could use wp_print_styles
4. wp_register_script / wp_enqueue_script only accepts 4 parameters.
5. You should only register your scripts and enqueue only the last one with its dependencies. Use the third parameter to define these.

Try out this code:

functions.php

add_action('wp_print_styles', 'wpse28490_enqueueStyles');
function wpse28490_enqueueStyles() {
    wp_register_style('fancyboxStyle', get_template_directory_uri() . '/css/fancybox.css');
    wp_enqueue_style('fancyboxStyle');
}

add_action('wp_enqueue_scripts', 'wpse28490_enqueueScripts');
function wpse28490_enqueueScripts() {
    if(!is_admin()) {
        // If our site is using SSL, we should load our external scripts also in HTTPS
        $protocol = (is_ssl()) ? 'https://' : 'http://';

        // jQuery
        wp_deregister_script('jquery');
        wp_register_script('jquery', $protocol . 'ajax.googleapis.com/ajax/libs/jquery/1.6.2/jquery.min.js', array(), '1.6.2', true);

        // Custom scripts
        wp_register_script('jquery-easing', get_template_directory_uri() . '/js/jquery.easing-1.3.pack.js', array('jquery'), '1.0', true);  
        wp_register_script('jquery-mouseWheel', get_template_directory_uri() . '/js/jquery.mousewheel-3.0.4.pack.js', array('jquery'), '1.0', true);
        wp_register_script('jquery-fancybox', get_template_directory_uri() . '/js/jquery.fancybox-1.3.4.pack.js', array('jquery'), '1.0', true);
        // fancyboxControls needs jquery, jquery-mouseWheel and jquery-fancybox, so we could pass these ids to the third parameter as an array.
        wp_register_script('fancyboxControls', get_template_directory_uri() . '/js/fancybox/fancyboxControls.js', array('jquery', 'jquery-easing', 'jquery-mouseWheel', 'jquery-fancybox'), '1.0', true);

        // Everything is registred, just enqueue our last script
        wp_enqueue_script('fancyboxControls');
    }
}