This list can be modified using WooCommerce filters. However, without a thorough testing I could not find a specific filter to help you. You might check the API for a better answer.
Meanwhile, I have used this CSS snippet to successfully hide the “All Categories” entry from the drop-down selector:
li.option[data-value=""] {
display: none;
}
This will only prevent users from selecting “All Categories” again. It will not prevent your page from defaulting to “All Categories”.
Hope this is helpful!