You can use both:
- Comment meta for the plain vote count, because that is easy to fetch with a standard query.
- A custom table storing additional data to prevent vote fraud and other issues. These tables might be used for rare events (voting) only, but they are not used to list comments in the back-end or in views where voting is not possible anyway.
So plan performance bottle-necks first, profile, then decide what setup works best for you.