This isn’t possible via register_block_pattern_category or filter priorities.
Gutenberg and the REST API at the time of writing don’t attempt to sort the categories, and instead display them in the order given to them by the Pattern repository class in PHP.
That pattern repository class stores them in an array as objects on keys, so they are not numerically indexed and sortable in the traditional sense. But even if they were, that array is private and unmodifiable:
private $registered_categories = array();
And the class is final:
final class WP_Block_Pattern_Categories_Registry {
And referenced directly by name, so it can’t be swapped out.
The best and only remaining hope is to use REST API filters to re-sort the JSON response how you prefer, but this is unlikely to bring it to the very top, this is because REST API pattern categories are always second after an internal list of categories registered via javascript.
This is why featured/posts/text/gallery appear first followed by the rest.