how to check if a shortcode is used more than one time in the same post

In your shortcode callback myshortcode_handler(), use a static variable:

function myshortcode_handler()
{
    static $first_call = TRUE;

    if ( ! $first_call )
    {
        # callback was called earlier …
    }

    # set it to FALSE after you have handled the condition.
    $first_call = FALSE;

    return;
}

Please do not misuse the global namespace for this: avoid constants, global variables and similar workarounds that might lead to unexpected conflicts.