A custom RSS feed link, how do I modify the XML structure?

All of the native feeds and those that are added in line with WP API (add_feed()) rely on feed=[type] query variable and corresponding do_feed_{$feed} action hook where they are executed.

From your description it’s different. Someone registered entirely different query variable in code and attached their own logic to it.

There is no way to say how precisely that was done without finding and reading code responsible (searching for customfeed would be a start).