Try
type: 'listbox',
values: [{text: 'small'}, {text: 'normal'}, {text: 'large'}]
For details: https://github.com/tinymce/tinymce/blob/master/js/tinymce/plugins/link/plugin.js
I met the same problem when I want to use select2 in the dialog. All plugins are using listbox. My guess is they always use listbox instead of selectbox. And the handler for selectbox is broken.