You can do this via JavaScript rather easily. Here are the steps you need to take. I’ve created a sample (untested) script below as a proof-of-concept for you.

Register your meta box

add_action('add_meta_boxes', 'foo_page_template_meta_box');
function foo_page_template_meta_box(){
    add_meta_box(
        'foo_page_template_options',
        __('Foo Page Template Options'),
        'foo_page_template_options',
        'page'
    );
}

function foo_page_template_options(){
    //Your custom field form goes here
}

Add JavaScript and CSS to admin_head for post.php and post-new.php

add_action('admin_head-post.php', 'foo_page_template_script');
add_action('admin_head-post-new.php', 'foo_page_template_script');

function foo_page_template_script(){
    global $post;

    # Only use this script for pages
    if('page' !== $post->post_type)
        return;

    #Name of page template file
    $file="foo_page_template.php";

    $output = "
        <!-- Hide the Meta Box -->
        <style type="text/css">
            #foo_page_template_options{display:none;}
        </style>

        <script type="text/javascript">
            jQuery(document).ready(function($){
                var _file="$file";
                var _meta_box = $('#foo_page_template_options');
                var _page_template = $('#page-template');

                //Show meta box on page load
                if(_file == _page_template.val())
                    _meta_box.show();

                //Sniff changes to the page template dropdown
                _page_template.change(function(){
                    if(_file == _page_template.val())
                        _meta_box.show();
                });
            });
        </script>
    ";
    echo $output;
}

The main premise of this approach is that you load the meta box on all ‘page’ edit screens, but only show it when the page template select value matches your page template file name.

Hope this helps!