Try to use wp_dropdown_pages instead of wp_list_pages .
This function does almost the same, but returns a dropdown, instead of list items. (do not need to echo select opening and closing tags separately)
if ( $post->post_parent ) {
$children = wp_dropdown_pages( array(
'name' => 'your-name' // paste field name here
'child_of' => $post->post_parent,
'post_type' => 'mycustom',
'echo' => 0
));
} else {
$children = wp_dropdown_pages( array(
'name' => 'your-name' // paste field name here
'child_of' => $post->ID,
'post_type' => 'mycustom',
'echo' => 0
));
}
if( $children ):
echo $children;
endif;
Hope you already know that select is a form element and it keeps post id as option value, so you need to solve this manually if you want to redirect users to selected page id. (Code is not tested)