Is there a way to ensure plugin script loads before another script?

Shortcodes are parsed while outputting content, so the problem may be that other scripts are always included first because hooks like wp_head() are called before WP ever gets to any content.

Instead of shortcodes, you could enqueue on the wp_enqueue_scripts hook, if you can find a query condition to isolate the places where you want the script enqueued:

<?php
add_action('wp_enqueue_scripts', 'wpse_first_enqueue', 1);
function wpse_first_enqueue() {
    // Only enqueue on your desired condition
    // In this example, only on individual Posts
    if(is_singular('post')) {
        wp_enqueue_script('another-custom-script', plugin_dir_url( __FILE__ ) . '/another-custom-script.js');
    }
}
?>

If you’re planning to distribute the plugin, and there isn’t a query condition that always clearly isolates the places where the script needs to fire, you might consider always enqueuing it on the front end with this low priority. You could still have users enter a shortcode which creates a div, and then in the JS, if that div exists in the content, fire your functionality. (If the div doesn’t exist, don’t fire anything.)

Or, if you’re just using this for your own site, and all of the JS files are enqueued properly, you could dequeue all scripts and then re-enqueue everything with dependencies. You can’t say “enqueue this one first,” but you can say “make sure Script A is enqueued before this next script is enqueued.” The downside is you will need to enqueue your script on every front end page, but again you can set up a condition that waits and only fires when someone has used the plugin on that page.

<?php
add_action('wp_enqueue_scripts', 'wpse_dequeue_and_re_enqueue', 1);
function wpse_dequeue_and_re_enqueue() {
    // Step 1: Dequeue jQuery
    wp_dequeue_script('jquery');
    // (continue dequeueing all JS scripts)
    // Step 2: Enqueue your script
    wp_enqueue_script('another-custom-script', plugin_dir_url( __FILE__ ) . '/another-custom-script.js');
    // Step 3: Re-enqueue everything, with your script as dependency
    wp_enqueue_script('jquery', '/wp-includes/js/jquery/jquery.js', array('another-custom-script');
    // (continue re-enqueueing all scripts)
}
?>

You’ll need to be careful about the order and dependencies, and it may be tedious to maintain, but this will guarantee that (with the current theme and plugin set) your script is always included before anything else.