Yes, the Plugin approval team is more than right, this is not the way to do it.

First a couple of articles from a core developer and an excellent plugin developer, where you’ll get to know why not to use wp-load.php:

• Don’t include wp-load, please, from Otto on WordPress

• wp-load.php – I Will Find You!, from Pippin’g blog Crappy Code.

When initiating your plugin, you can define the plugin’s URL as follows:
$plugin_url = plugins_url( "https://wordpress.stackexchange.com/", __FILE__ );

And when calling your download script, you can pass the needed information in a query var, e.g., $plugin_url . 'download.php?file=FILE-URL' and $_GET it in your script.

This is just a general outline, more details on your plugin’s logic would need to be provided if this outline is not enough. And, yes, as @AJHenderson points out, WordPress Answers is the place where you’ll find lots of developers, including Otto and Pippin.

Security notice

You need to have your download script tightly secured, as one could do the following:

http://example.com/path/to/plugin/download.php?file=http://example.com/wp-config.php