Why does WordPress append a bullet to my category tag?

Read the documentation for the_category():

Display category list for a post in either HTML list or custom format.

If the first argument isn’t given, or is '' then the categories will be output in a <ul>. To output them comma-separated, as an example, set the $separator to ', ':

<?php the_category( ', ' ); ?>

Every core function is documented in the Code Reference. If a function isn’t behaving as expected, check the documentation.