Prepending character(s) to a custom tag title?

Maybe I’m misunderstanding the question – if so apologies!

» 

represents the ยป symbol.

I think all you need to do is add it into your second example, if that’s what’s displaying in your template:

<?php the_content("&raquo; Continue reading " . the_title('', '', false)); ?>

Hope this helps!