It required the actual dns name, and MathJax.js as the filename:
http://ec2-[...].compute-1.amazonaws.com/wp-content/plugins/mathjax-latex/MathJax/MathJax.js
Requiring the dns name is unfortunate since that changes each time if I stop and later reactivate the VM.