Trying to add script to specific admin page is not working

I think you’re using the wrong approach. Rather than removing the submenu item, alter how the post type was registered, and change the show_in_menu argument. Hook into init way late and change the argument.

<?php
add_action('init', 'wpse99123_post_type_switcher', 999);
function wpse99123_post_type_switcher()
{
    global $wp_post_types;
    $wp_post_types['contact']->show_in_menu = true; // put it back in the menu
}

Then, to load the CSS/JS/whatever, you need to hook into load-edit.php and check the current screen’s post type and add enqueue functions from there.

<?php
add_action('load-edit.php', 'wpse99123_load_edit');
function wpse99123_load_edit()
{
    $screen = get_current_screen();

    if (!isset($screen->post_type) || 'contact' !== $screen->post_type) {
        return; // not where we want to be, bail
    }

    // add enqueues here.
    add_action('admin_enqueue_scripts', 'wpse99123_enqueue');
}

function wpse99123_enqueue()
{
    // ...
}

You have to do load-edit.php because $screen->post_type will be contact on your contacts edit (post.php) and add new (post-new.php) pages.