I have not a working solution at hand, but what I would do is to analyze how this is done for the more seperator. In the HTML editor, there is <!-- more -->, in the visual editor, an image is displayed instead.
This is done by extending the tinyMCE editor – which is the base of the visual editor in wordpress – with a plugin. To learn more about tinymce plugins, you find more examples and documentation in the moxiecode wiki: Creating a plugin for TinyMCE (3.x).
You find example code within the following file:
wp-includes/js/tinymce/plugins/wordpress/editor_plugin.dev.js
It works basically by replacing content (e.g. the shortcode [mycode]) with some predefined HTML that contains the image. Before the content get’s posted, it is replaced again with the original HTML. You can make use of regular expressions in the process. The more link is showing this quite well.