How do i use fontawesome icons in TinyMce editor?

You wouldn’t be able to add a FontAwesome icon by passing it directly to the ed.addButton(); method unfortunately.

You can try a workaround however. If you leave the image : url+'/youtube.png' parameter out of the method then it will automatically create an empty <span> with the class of mceIcon & another class of mce_[plugin_name].

You can then use CSS to style that <span> element however you would like.

You can use FontAwesome directly in your CSS now, something like this (make sure you change .mce_[plugin_name] place-hoder class used here to your actual class):

span.mce_[plugin_name] {
    position: relative;
}

span.mce_[plugin_name]:before {
    content: "\f166"; /* Value for the Youtube icon*/
    font-family: FontAwesome;
    font-style: normal;
    font-weight: normal;
    text-decoration: inherit;
/*--adjust as necessary--*/
    color: #000;
    font-size: 18px;
    padding-right: 0.5em;
    position: absolute;
    top: 10px;
    left: 0;
}

UPDATE

Here is how I load my FontAwesome CSS file. I have adapted to load for the admin area for what you need to do however. I call it from the BootStrap CDN, but you could download the CSS file and load it from your theme or plugin folder using the same admin_enqueue_scripts(); function.

// Load Font Awesome
function royal_enqueue_awesome() {
    wp_enqueue_style( 'prefix-font-awesome', '//netdna.bootstrapcdn.com/font-awesome/4.0.3/css/font-awesome.min.css', array(), '4.0.3' );
}

add_action( 'admin_enqueue_scripts', 'royal_enqueue_awesome' );