How to add a custom button to the tinyMCE toolbar?

Your problem i believe are the lines that follow the enqueues and print scripts, you’re mixing Javascript with PHP..

Javascript goes inside the HTML section of a PHP file or inside an echo statement.

This page of the codex gives an example for adding a button to TinyMCE inside WordPress.

However that codex entry might be a bit dated, so in the event of problems with the code here are some alternative code samples to work from.

http://brettterpstra.com/adding-a-tinymce-button/
https://stackoverflow.com/questions/4413470/how-to-add-a-mailto-button-to-tinymce

RE: Code problems

Inside a PHP file, HTML goes between the ending and starting PHP blocks, like so..

?>

<div>example</div>

<?php

Or alternatively inside an echo statement.

echo '<div>example</div>';

Javascript should be treated in the same manner as HTML, so you above code(the JS part) should look either like this..

?>
<script type="text/javascript">
setup : function(ed)
{
     // Add a custom button
     ed.addButton('mybutton', 
     {
          title : 'My button',
          image : 'img/example.gif',
          onclick : function() 
          {
          // Add you own code to execute something on click
          ed.focus();
          ed.selection.setContent('Hello world!');
     }
});
</script>
<?php

Or..

echo "
<script type=\"text/javascript\">
setup : function(ed)
{
     // Add a custom button
     ed.addButton('mybutton', 
     {
          title : 'My button',
          image : 'img/example.gif',
          onclick : function() 
          {
          // Add you own code to execute something on click
          ed.focus();
          ed.selection.setContent('Hello world!');
     }
});
</script>
";

The former being the easier approach in my opinion.

I hope that answers your question, and if you need help understanding something let me know..