Updates for a private plugin?

Looks like the applicable code is in wp-includes/update.php, wp_update_plugins():

$to_send = (object) compact('plugins', 'active');

$options = array(
    'timeout' => ( ( defined('DOING_CRON') && DOING_CRON ) ? 30 : 3), 
    'body' => array( 'plugins' => serialize( $to_send ) ),
    'user-agent' => 'WordPress/' . $wp_version . '; ' . get_bloginfo( 'url' )
);  

$raw_response = wp_remote_post('http://api.wordpress.org/plugins/update-check/1.0/', $options);

It specifically checks api.wordpress.org. Technically speaking it would be possible to pass a key inside $to_send to delegate the check, but to my knowledge that is not a supported feature.

If you hook into set_site_transient_update_plugins you could add your own package details into this variable. It looks like those values will be trusted when you run the plugin updater. See wp-admin/update.php and wp-admin/includes/class-wp-upgrader.php. Given the code in these two functions, I think it would be possible to inject your own update server, you just need to look at how the package details are formatted and match that.