Custom WP_Query for current category (in category.php)?

Try referencing the cat query variable like so:

$custom_query = new WP_Query( 
    array( 
        'cat' => get_query_var('cat') 
    ) 
);