How to customize the symbols that appear in the Visual Editor insert custom character

This has to do with your character map for tinyMCE it’s located inside:

wp-includes/js/tinymce/themes/advanced/js/charmap.js

Although I wouldn’t recommend editing the WP-Core you could theoretically just add new characters there like so:

['Ī',     'Ī',  true,'I kahako'],
['Ō',     'Ō',  true,'O kahako'],

and so on…

However when you update WordPress this will overwrite those changes. So it’s not the best solution but it would work.