You can use the following code in your category archive template file.
if (is_category()) {
global $query_string;
$post_type = get_query_var('post_type');
if ( !$post_type ) { // if no post type is already defined
parse_str( $query_string, $args );
$args['post_type'] = array( 'post', 'you-custom-post-type-here' );
query_posts( $args );
}
}
You can also use the above code in any action that gets run just before the main loop.