Proper way to use WordPress function with AJAX PHP file

Yes, using admin-ajax.php is the way to go here. To use it, you would do something like the following:

in vote_database_update.php

// hook into admin-ajax
// the text after 'wp_ajax_' and 'wp_ajax_no_priv_' in the add_action() calls
// that follow is what you will use as the value of data.action in the ajax
// call in your JS

// if the ajax call will be made from JS executed when user is logged into WP,
// then use this version
add_action ('wp_ajax_call_your_function', 'your_function') ;
// if the ajax call will be made from JS executed when no user is logged into WP,
// then use this version
add_action ('wp_ajax_nopriv_call_your_function', 'your_function') ;

function
your_function ()
{
    if (!isset ($_REQUEST['id'])) {
        // set the return value you want on error
        // return value can be ANY data type (e.g., array())
        $return_value="your error message" ;

        wp_send_json_error ($return_value) ;
        }

    $id = intval ($_REQUEST['id']) ;
    // do processing you want based on $id

    // set the return value you want on success
    // return value can be ANY data type (e.g., array())
    $return_value="your success message/data" ;

    wp_send_json_success ($return_value) ;
}

in custom-js.js

(function ($) {
    $('.product-vote-button-up').click (function () {
        var productID = $(this).attr ('productID') ;

        $.ajax ({
            url: '/wp-admin/admin-ajax.php',
            type: 'POST',
            dataType: 'JSON',
            data: {
                // the value of data.action is the part AFTER 'wp_ajax_' in
                // the add_action ('wp_ajax_xxx', 'yyy') in the PHP above
                action: 'call_your_function',
                // ANY other properties of data are passed to your_function()
                // in the PHP global $_REQUEST (or $_POST in this case)
                id : productID
                },
            success: function (resp) {
                if (resp.success) {
                    // if you wanted to use the return value you set 
                    // in your_function(), you would do so via
                    // resp.data, but in this case I guess you don't
                    // need it
                    $('#product-' + productID + ' .item-product-footer-vote-container').html ('Thanks for your vote!') ;
                    }
                else {
                    // this "error" case means the ajax call, itself, succeeded, but the function
                    // called returned an error condition
                    alert ('Error: ' + resp.data) ;
                    }
                },
            error: function (xhr, ajaxOptions, thrownError) {
                // this error case means that the ajax call, itself, failed, e.g., a syntax error
                // in your_function()
                alert ('Request failed: ' + thrownError.message) ;
                },
            }) ;
        }) ;
    })(jQuery) ;

I hope this helps.