Is there a limit to hook priority?

There are no limits and no performance penalties. To understand why, you need to understand how all hooks are stored in the WP ecosystem.

First of all you need to understand where all hooks are stored and how they do it. All hooks for filters and actions are stored in global variable called wp_filter, yes yes action hooks are stored in this variable too. This variable is associated array, where key is the name of action or filter and value is another associative array. For example let’s take a look at ‘init’ action, at this stage we will see following structure:

$wp_filter = array(
    'init' => array(...),
);

This sub array has numeric keys and values as arrays. Numeric keys are our priorities. Arrays, associated with numeric keys, contain a list of hooks with the same priority. So if we call add_action( 'init', 'wpse8170_my_first_init', 20 ), then call add_action( 'init', 'wpse8170_my_second_init', 20 ) and finally call add_action( 'init', 'wpse8170_my_third_init', 10 ), our example will look like:

$wp_filter = array(
    'init' => array(
        20 => array(
            'wpse8170_my_first_init' => array(
                'accepted_args' => 1, // the number of accepted arguments by your hook
                'function' => 'wpse8170_my_first_init', // callback function
            ),
            'wpse8170_my_second_init' => array(...),
        ),
        10 => array(
            'wpse8170_my_third_init' => array(...),
        ),
    ),
);

Now when init action is triggered all hooks will be sorted with usage of ksort function and our array looks now:

    array(
        10 => array(
            'wpse8170_my_third_init' => array(...),
        ),
        20 => array(
            'wpse8170_my_first_init' => array(
                'accepted_args' => 1, // the number of accepted arguments by your hook
                'function' => 'wpse8170_my_first_init', // callback function
            ),
            'wpse8170_my_second_init' => array(...),
        ),
    ),

And all hooks will be executed in this queue: first 'wpse8170_my_third_init', then 'wpse8170_my_first_init' and finally 'wpse8170_my_second_init'.

So you can see that there is no limits and penalties and you can use any value which is acceptable as a key for associated array by your PHP environment.