Can I specify that custom.css gets loaded at Appearance > Editor instead of style.css?

This seems to be controlled by global $file variable, which is filled from request (GET or POST) by WordPress. So you will need to pass file you want in request or hook somewhere and override value of the variable. If latter be careful that you don’t lock editing to single file altogether.

See source for details.