How can I troubleshoot why TinyMCE won’t load custom plugins in the visual editor?

You should read this: http://codex.wordpress.org/TinyMCE_Custom_Buttons

It’s pretty straightforward if you already know how to program a TinyMCE plugin (which it sounds like you do).