What you are seeing, I believe, is a result of paginate_links using three blocks of numbering– first, middle, and last. If you had enough pages you’d get, for example, when browsing page 50 …
1 2 3 … 49 50 51 … 98 99 100
If you imagine the “middle” numbers overlapping the “first” numbers you’d get 3 or 4 or five numbers bunched up. You can play with the end_size and mid_size parameters and try to get something you like better, or use 'type' => 'array' and write your own display function to convert the resulting array into actual pagination links.