How can I use this code on a custom database table?

Yes you have to write a custom function and call it on your manage columns it will help you to get the necessary contents.

     public function xxxx_get_cpt_data($post_ID) {
          //        $cpt_values = get_post_custom($post_ID);
           global $wpdb;

             $sql = "SELECT * FROM ". $wpdb->prefix. "custom_table WHERE post_ID=". $post_ID. " LIMIT 1" ;
              $cpt_values = $wpdb->get_results($sql);
            if ($cpt_values) {
                  return $cpt_values->column_name;
             }else {
                  return ' ';  
             }
       }

But the get_results returns an array of objects. If you wish to get a single result than use get_var.
If you have multiple values on the custom table go with get_results use it like this $obj->coulmn_name;