DOM reference to TinyMCE editor element (button)

Try adding an id attribute in your button definition:

    ed.addButton('solution', {
        disabled: 0,//count_problems(ed).length == 0,
        id: 'my_button',
        title : 'Add problem solution',
        //cmd: 'solution_fun',
        image : url + '/images/wspringer.png',
        type: 'menubutton',
        menu: [{text: 'problem id = ' + 1, value: 1}],
        onselect: function(v1) {
            ed.windowManager.open({
                title: 'Insert solution in popeye output format',
                body: [{type: 'textbox', name: 'solution', label: "solution", width: 300, height: 50}],
                width: 270,
                height: 70,
                onsubmit: function(v2) {
                    console.log(v1);
                    solution( v2.data.solution, v1.control.settings.value );
                }
            });
        };
    });

And see if you can access is with jQuery('#my_button')