How to keep track of when a vote is cast?

The following lines of code both store the IP address and time of vote,

// $ip holds $_SERVER['REMOTE_ADDR']
$voted_IP[$ip] = time();  
// Save IP and increase votes count  
update_post_meta($post_id, "voted_IP", $voted_IP);

The time a vote is cast is already being stored for you in the post_meta table for the given post the person is voting on.

To retrieve the votes and corresponding times for a given post you can do this,

  $mykey_values = get_post_custom_values('voted_IP', $post_id);
  foreach ( $mykey_values as $key => $value ) {
    echo "$key  => $value <br />"; 
  }

That would return a list in a similar format to,

192.166.342.8 => 1349260674
202.646.100.3 => 1349260674
^ IP numbers     ^ unix time stamps